Steps to reproduce:
1. Open the attached project
2. Click the 'Wall' prefab
3. In the Inspector tab go down to Asset Labels
4. Add a label 'Wall' to the prefab and Save project
5. Locate the prefab in your OS' File Explorer
6. Open the 'Wall.prefab.meta' file with a text editor
7. Remove the lines 'labels: - Wall ' and save
8. Check back in Unity Editor and notice that the prefab still has the label 'Wall'

Note: After restarting Unity, the label still persists, while the .meta file contains no labels

Workaround: Library folder needs to be deleted to update Asset changes