Reproduction steps:
1. Create a new project;
2. Create a new resolution 6400 x 5800 in Game window;
3. Observe Console window;

Actual result: the Game view is reduced to a larger resolution than the one you set. For this (6400 x 5800) resolution the warning is "GameView reduced to a reasonable size for this system 6249 x 6895)".

Expected result: the Game view should be reduced to a smaller resolution if it is too large. For this (6400 x 5800) resolution the warning should be "GameView reduced to a reasonable size for this system 6101 x 5529)".

Reproduced with: 5.6.0a1, 5.6.3f1, 2017.1.0p3, 2017.2.0b5, 2017.3.0a2
Not reproducible with: 5.4.6f1, 5.5.0b1, 5.5.4p2
Regression introduced in 5.6.0a1

Note: the bug appears with this (6400 x 5800) resolution if a computer is with about 2GB of video memory. If a machine is with 4GB or more of graphics memory, the issue should not appear with that resolution.

Verified with 2017.3.0a4