To Reproduce:
1. Open 'reproProj.zip' attached to the post below ( use the project attached by the user if using Windows).
2. Run, observe that 'simpleTestPlugin.bundle' is loaded, exit play mode, remove 'simpleTestPlugin.bundle' from Plugins dir.
3. Play, click 'PrintNumber..' button, observe that the plugin is still loaded even though it no longer exists in the project.

4. This means that native plugins cannot be updated without restarting the editor. And also this behavior is inconsistent (all changes done in play mode should be removed when gong back to editor) .

It might be possible to unload DLLs manually in 'OnApplicationQuit' using 'FreeLibrary' in Windows. But no equivalent for this function seems to exists on OSX.

This is a Net/Mono issue, but if is there is a way to work around this it would be nice if Unity unloaded native plugins when exiting play mode.