To reproduce:

1. Open attached project
2. Open the profiler
3. Open the scene "Scene_With_Issue"
4. Play the scene
5. Press one of the UI buttons
6. Note how much FPS the scene runs at
7. Open the scene "Scene_Without_Issue"
8. Play the scene
9. Press the same button as in 5
Notice that the FPS is now much bigger, even though the only difference is that the problem scene sets the color values through animation, and the scene without issue sets a dummy object's coordinates through animation and then copies them to the color values via script. On my machine the numbers are 30-100 FPS on the scene with issue and 200FPS+ on the scene without issue, with the exact numbers depending on the Unity version.