If you're using an MSI device this might fix your problem: http://answers.unity3d.com/questions/1153791/lag-in-scene-view-1.html

Do you have Nahimic for MSI installed? That was the culprit in my case. Uninstall that shit and restart Unity, might work for you too. If it doesn't work, make sure there's not a service for it still running, also try rebooting after uninstalling it.

Unbelievable that this is still not fixed. Postponed and voting is disabled. What a crock, Unity. At least give us the code so we can fix it if you don't want to.

updatE: still stuttering

So is there any solution ? Now upgraded to unity 5.5 and the bug still persists, one thing that helps is when you run youtube.com in background the all of a sudden the sutter stops... But i want to fix that completely so i don't have to open chrome every time

I've got the same issue now in 5.3.4 and the workaround I mentioned earlier working does not help anymore. Please look into this.

To be honest, I'm not having this issue anymore since 5.4.0b<something>, it stutters a little when you just launch the editor the it stops stuttering and delaying.

Just repeating that Animated Materials in the skybox toggle at the top of Scene window fixes the WASD stuttering, but not the switching of direction stuttering.

You would think that after all this time this would be fixed...

I actually created a new empty project, cleared all editor tabs with the exception of the scene view, turned off the sun and the skybox box, opened unity with both the integrated graphics and my gtx980m. Nothing helped. Tested in both the lasted stable 5.3.4f1 and 5.4b11. Same behavior.

Unity, you already acknowledge in GDC you guys have been introducing too many bugs. How about start doing what you promised and correct things?

I was getting this (around 1s freezes when switching panning directions in the editor) on Unity 5.3.3. I don't think it began right after upgrading Unity, but at a seemingly random point of using the editor. Then it was occurring on all my scenes, trivial ones included.

Disabling animated materials as suggested did not change things for me. But another toggle did, the scene lighting one, represented as a sun icon in the scene view. After disabling that, panning delays were gone. But after enabling lighting again, delays did not return.

I'm anticipating the problem will keep occurring, but I've got no clue what could be causing it. Somebody please take a look at this. Meanwhile try my workaround, if the other one mentioned didn't work for you.

Can't you just temporarily add some toggable code that will constantly move the camera in scene view by some fractional amount while you look for a fix? cmon Unity, its been months... please...