How to reproduce:
1. Download attached project files and open project
2. Drag the prefab "Crash Demo" into an empty scene and select it.
3. Click the "Set Custom Instance Modifications (Causes crash or freeze)" button.
4. Observe crash or freeze - it can happen instantly or after a short moment.
4.1 If crash or freeze does not happen, close the project (do not save the scene) and repeat the steps again

Actual result: Whenever custom prefab modifications are manually set using PrefabUtility.SetPropertyModifications, that reference certain Unity objects (and only certain Unity objects - with others it works fine), either 1) an internal Unity crash will happen, or 2) Unity will freeze indefinitely, taking up a full core's worth of CPU time. This happens on both Windows and Mac

Reproduced with: 5.5.2f, 5.6.0f3, 5.6.1f1, 5.6.3p1, 2017.1.0p4, 2017.2.0b8, 2017.3.0a5

Note: Observe that the crash does not happen if you drag the "No Crash Demo" prefab into the scene and click the button. As far as we can tell, this is because it does not contain any UnityEngine.UI related components.