Steps to reproduce:
1. Open Unity
2. Create new project or open any other existing project
3. Create GameObject in the hierarchy
4. Try to close Unity, it should ask if you want to save changes click cancel
5. Try to close Unity again

Actual result:
Quit action not responding from the Editor. See the attached video "quitting.mp4".

Expected result:
Clicking Unity -> Quit button or COMMAND+Q or right click on Unity icon and selecting Quit should ask if you want to save changes before quitting.

Workaround:
Just before doing this, if you need your changes saved, please save first, because this workaround will not ask you if you want to save your changes and close Unity immediately. After that when you are ready to quit Unity just click Apple icon -> Force Quit -> Unity should be selected in the opened window and click Force Quit.

Other notes:
The bug is not reproduced on Windows operating system.

Reproduced with: 5.4.0b10
Not reproduced: 5.1.4f1, 5.2.4f1, 5.3.3p2