Steps to repro:

1. Open Unity on OSX;
2. Create new project;
3. Go to Player Settings (Edit>Project Settings>Player);
4. Uncheck 'Auto Graphics API for Mac' checkbox;
5. Drag 'OpenGLCore' on top of the list;
7. In the opened popup window click 'Cancel' button.

Expected result:
No errors are thrown in the Console.

Actual result:

2 errors:

NullReferenceException: Object reference not set to an instance of an object
UnityEngine.GUILayoutUtility.BeginLayoutGroup (UnityEngine.GUIStyle style, UnityEngine.GUILayoutOption[] options, System.Type layoutType) (at /Users/builduser/buildslave/unity/build/Runtime/IMGUI/Managed/GUILayoutUtility.cs:296)

and

InvalidOperationException: Operation is not valid due to the current state of the object
System.Collections.Stack.Pop () (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections/Stack.cs:329)

are shown in the Console. See attached video.

Regression introduced in Unity 2017.3.0.

Notes:
- Reproducible in 2018.1.0a5, 2017.3.0b10;
- Not reproducible in 2017.2.0p1, 2017.1.0p4;
- Reproducible on OSX 10.13.1; not reproducible on Windows 10.