i have also experienced this exact same issue using a custom OnAudioFilterRead setup with a generative synth plugin in Unity. i had previously had good results with Best Latency as well, on an older game using Unity 5.3. recently switched to 2017 for this new generative audio project and the audio performance was terrible under Best Latency - unusable. i did assume that the graphics performance and particle effects might be causing the glitching, however. i switched to Best Performance and the results are smoother but timing is a bit rougher. reading this it seems like we have a one step forward, two steps back situation and as always, audio gets the short end of the stick.

There are severe bugs in the Unity audio implementation, see

https://issuetracker.unity3d.com/issues/audio-dropouts-are-caused-by-garbage-collector-when-large-amount-of-memory-are-being-allocated

Because of this flaw it will be impossible to get glitchless audio in any other than the most trivial applications that use very little CPU. And this is on best performance setting. I believe the low latency setting will work only by chance.

This issue is a problem for my projects too! Please fix this Unity so I can update to the latest version.

Like Thelo, I encounter this issue only on Windows, but not on Mac end Linux. Setting the Buffer size on anything else than "Best Latency" results in a perceptible input latency. One way or the other, developing rhythm games becomes a lot more difficult.

I'm requesting that the issue be reopened. For the benefit of others following this issue, here's the response received from QA:
" Hey there!

I have just received a reply from a developer. He says:
"Best latency means that buffer sizes are smaller and there are higher chances of not being able to service the sample card in time.

This is a basic relationship between latency and performance, especially on games that are already pushing the CPU.

If the user is experiencing glitching, its because the buffering isnt large enough and they should choose a different setting.

Its kind of like changing the resolution of a game to 4K and then wondering why the framerate drops to 5fps."

If you have any questions, please write me back.

Best regards,
----
QA Team"

And my response to that:
"Hey ----,

I appreciate your help, but unfortunately the response from your developer is frustrating for being dismissive and condescending.

[quote]This is a basic relationship between latency and performance, especially on games that are already pushing the CPU.[/quote]
The relationship between smaller buffers and higher CPU requirements is understood.

[quote]If the user is experiencing glitching, its because the buffering isnt large enough and they should choose a different setting.[/quote]
What I demonstrated with the test project I attached to the bug report is that with exactly the same settings, 5.3.5 does not exhibit audio glitching, but later editor versions do.
All things being equal besides the editor version, the audio system exhibits strain at the best latency setting in versions after 5.3.5.

Our problem may be an edge case, but it's a big one since it means our project is stuck at version 5.3.5. Moving to any later editor version causes the audio to bug out. Our project is a VR audio creation tool that absolutely depends on low latency - doing things like recording drumming against a metronome is impossible at any setting but best latency.

I request that the issue tracker case be reopened - preferably with the original wording of my bug report intact, since the revised wording creates the impression that it's an editor problem only and fails to mention that it also occurs in built projects.

Thanks again for your help."

Also, here's the related forum thread:

https://forum.unity3d.com/threads/heavy-audio-stutter-in-5-4-but-not-5-3.436664/

I'd like to add that I've seen this problem happen on Windows, but not on Mac.

Just to add, the issue also occurs in built projects set to best latency, not only when switching latency settings in the editor.