Reproduction steps:

# Open the user’s attached project “ReproProject”
# Open the “_Assets/Test.unity_” scene
# Navigate to “_Preferences/General/Script Changes While Playin_g” and set it to “Recompile After Finished Playing”
# Navigate to “_Preferences/Asset Pipeline/Auto Refresh_” and set it to _Enabled_
# Enter Play Mode
# Open the “_Assets/Test.cs_” script, uncomment line 11
# Save the script and return back to the Editor

Expected result: Script does not recompile in Play Mode
Actual result: The script recompiles while still in Play Mode

Reproducible with: 2021.1.0a8, 2021.1.28f1, 2021.3.15f1, 2022.2.0f1, 2023.1.0a20
Not Reproducible with: 2020.3.42f1, 2021.1.0a7
Reproducible on: Intel macOS 13.0.1

Note: Changing the Auto Refresh selection to “Enabled Outside Play Mode” seems to stop the recompilation in Play Mode.