This is by design because when auto mode is used for building the lighting, editor fetches the data after quitting play mode. Therefore we see the progress bar in the editor.

To reproduce:
1. Open the attached project/create a new project
2. Press "play", press it again to stop
3. Notice that the lights rebake

Note: with a new project or the attached one the time is short enough that it might be hard to actually notice it, look for the blue progress bar in the corner of the screen

Reproduced on: 5.6.0a5, 5.6.0a3, 5.4.2p2