Assuming you have a Collab enabled project. You have done the first publish. A folder containing scripts and assets exist on your project. You have an object on your scene.

1. Delete the folder (here's a link - https://oc.unity3d.com/index.php/s/MayDpbegS7VvOek - to the folder I used. Let me know if you have access).
2. Press Play on your scene

Notice that the Play button will remain in a pressed state without ever going into "Play" mode. It will remain this way for a short amount of time IF the user refreshes or creates another asset or restores history/goes back in history. This suspended state only happens when the Editor is compiling scripts.

It will NOT happen if you don't have an object on the scene. It will NOT happen if the Editor is not compiling scripts.