1. Start a new project
2. Navigate to tab Edit->Preferences: General and turn off auto-refresh settings
3. Navigate to tab GameObject ->3D object (can probably be anything in fact) and place it in your scene.
4. Right click on the project view ->Create ->C# script.
5. In Scene view, left click your GameObject, select the inspector view.
6. In the inspector view, add a component to your gameobject, select script from the list and attach your newly created script.
7. Navigate to tab File ->Save Scenes and save your scene at a spot of your choice
8. Attach a behavior to your script (in my case I added "Debug.Log ("This is an error message");". Save your script.
9. Enable Collab
10. Press Play within the next 2-3 seconds

Observe that the Play button remains grayed out and cannot be unclicked. The user is otherwise stuck in "Play" until he quits Unity or creates a new asset.

Reproduced on: 5.5.0f3, 5.6.0f1, 2017.1.0a2