How to reproduce:
1. Open the attached project “IN-132774”
2. Open the “SampleScene”
3. In the Hierarchy, select the “TEST_OBJECT” GameObject
4. In the Animation window, move a few frames forward
5. In the Inspector window, change rotation values on all axes to 10
6. In the Animation window, add keyframes for the new values
7. Observe the Console and the animation in Scene View

Actual result: “Assertion failed on expression: 'CompareApproximately(SqrMagnitude(result), 1.0F)'“ is thrown in the Console and rotation values on all axes are set to -7.730941e+11 on all frames before the added keyframe
Expected result: No errors are thrown and rotation is animated

Reproducible with: 6000.3.0a1, 6000.3.7f1, 6000.4.0b7, 6000.5.0a6
Not reproducible with: 6000.0.67f1, 6000.2.0b2 (2bdac9ac0d74)

Reproducible on: Windows 11
Not reproducible on: No other environments tested

Workaround: Adding new keyframes on frame 1 and deleting the automatically added keyframes

Note: Issue appears to happen when transitioning from the keyframes that are added automatically when adding the Transform property to the next added keyframes