Reproduction steps:
1. Open attached project (ReproProject.zip)
2. Open and play the "TestScene" scene
3. Scroll trough the elements using mouse wheel
4. Notice that scrolling stutters
5. In hierarchy select the "Content" gameObject located under Canvas -> Panel -> ViewPort -> Content and enable Image component
6. Repeat step 3 and notice that scrolling doesn't stutter

Actual result: Scrolling with mouse wheel stutters

Expected result: Scrolling with mouse wheel shouldn't stutter

Reproduced on: 5.1.4p1, 5.2.4p1, 5.3.2p1, 5.4.0b3