To Reproduce:

1. Click build in iOS build settings.
2. Enter any new folder name.
3. Click save.
4. The build fails (for some reason).

Observe that the destination folder is still created, this should not happen if the build fails (not a big deal, but slightly annoying in some cases...)

You need the build to fail for some reason, that east to do, you can copy paste some of the dll's in Unity.app/Contents/UnityExtensions to get something like this:

Plugin 'OVRPlugin.dll' is used from several locations:
/Users/paulius/Desktop/UnityBuilds/Trunk_dec_1/Unity.app/Contents/UnityExtensions/Unity/VR/Win64/OVRPlugin.dll would be copied to <PluginPath>/OVRPlugin.dll
/Users/paulius/Desktop/UnityBuilds/Trunk_dec_1/Unity.app/Contents/UnityExtensions/Unity/VR/Win/OVRPlugin.dll would be copied to <PluginPath>/OVRPlugin.dll
Please fix plugin settings and try again.

Or do something else..

Not a regression, reproducible with 5.3 & 5.4 (at least)