To reproduce:
1. Open attached project ("case_1216702.zip")
2. From Hierarchy select "TimelineControl" Game Object
3. Press Play and in Timeline window select the Control Track
4. Observe the FPS before and after the Control Track enters a "HOLD" phase

Expected result: The FPS remains relatively close
Actual result: There is a significant FPS drop

Reproducible in: 2017.4.36f1, 2018.4.17f1, 2019.2.20f1, 2019.3.0f6, 2020.1.0a22

Reproducible in package versions: 1.1.0, 1.3.0-preview.6

Note: On a testing machine, the FPS drop resulted in a ~40% FPS loss (from ~2450 FPS while playing the initial clip, down to a ~1500 FPS while in the "HOLD" phase).