Steps to reproduce:

1. Open attached project "790680.zip"
2. Open scene "repro"
3. Open profiler
4. Run it
5. Once Ball touches terrain, press w to move it. notice that spikes are near 4ms

Note: Object contains A LOT of child components that aren't physics related and have Mesh Renderer component attached. Also, removing Mesh Renderer component from child game object decreases spikes.

Additional steps:
6. Open scene "norepro"
7. Run it
8. Press w after 4~ seconds
9. Notice that now physics take around 1ms

Reproduced with: 5.2.4f1, 5.3.4p5, 5.4.0b17