To reproduce:
1. Load the MicTest scene and enter Play mode.
2. Notice that the sphere's scale responds to microphone input; it scales itself according to the root mean squared of the sample data. Notice that the sphere is quite responsive when first entering play mode.
3. Tab to another application to take focus away from the editor. Tab back to the editor after a few seconds and notice that a significant amount of latency has been introduced into the microphone input processing.

Closed as By Design with this response:
We can't fix this. The best thing to do is to reset AudioSource.time in the OnApplicationFocus(bool focusStatus) callback. You need to do this anyway at startup to avoid the recording position to be at a random place relative to the play position.
When you do this you must also take into account the buffering happening between the microphone writer and the AudioSource reader. An empirical method that I've found works well is to offset the read position such that it is delayed by 3 DSP buffer sizes relative from the microphone position, like this:

int dspBufferSize, dspNumBuffers;
AudioSettings.GetDSPBufferSize(out dspBufferSize, out dspNumBuffers);

int mikeSeconds = 1;
int mikeSamples = AudioSettings.outputSampleRate * mikeSeconds;

source.clip = Microphone.Start(null, true, mikeSeconds, AudioSettings.outputSampleRate);
source.Play();

source.timeSamples = (Microphone.GetPosition(device) + AudioSettings.outputSampleRate - 3 * dspBufferSize * dspNumBuffers) % mikeSamples;

Notice that this is done after the microphone has been attached and the source has started playing, and notice the wrapping around mikeSamples.