How to reproduce:
1. Open the attached Unity project (jitters.zip)
2. Open the "test" scene
3. Select the Cube. In the Animation or Timeline window, set the slider to the 139th frame, press Record
4. Slowly slide the Cube along the X axis until jitters occur, place the Cube somewhere around X position value -0.6
5. Rotate the Cube along the Y axis using the Scene View gizmo until significant displacement occurs. Stop rotating the Cube and scale it in all axes at once
6. Move the Timeline or Animation slider and observe the Keyframe being placed in one frame lower than supposed

Expected result: while recording, the GameObject can be transformed without jitters. Misplaced Keyframes do not occur when added automatically.
Actual result: while recording, the GameObject jitters to different axes while transforming it. Keyframes added automatically are misplaced.

Reproducible with: 2017.1.0b6, 2017.1.3p2, 2017.2.2p1, 2017.3.2f1, 2017.4.0f1, 2018.1.0b13, 2018.2.0a6
Not reproducible with: 2017.1.0b1, 2017.1.0b5
Could not test with: all 5.6 versions and versions 2017.1.0a1 to 2017.1.0a6, because the Timeline wasn't introduced yet.

Notes: using the Move, Scale, Rotate and Rect tools to transform the GameObject seems to reproduce the issue. The jitters seem to occur randomly only whenever the Timeline or Animation slider is in some seemingly random positions - having more Keyframes seems to increase the reproduction rate.
Only Keyframes that are added automatically are misplaced. Adding the Keyframes with the "Add Keyframe" button or by double-clicking places the Keyframes in the correct position.
Animation clips which are not converted from Infinite clips don't suffer these issues.