To reproduce:
1. Open attached project
2. Select video in project window
3. Enable transcode and set Custom dimensions to 4000x4000 or higher and set Spatial Quality to High Spatial Quality
4. Apply changes
5. Drag video to hierarchy and run scene

Actual: Player fails to load with following errors:

mp4 video:
Player failed to load "file:///Users/audrius/Documents/New%20Unity%20Project%2012/library/metadata/30/30faa55cb1bad4f989c4794dc9dc057b.resource"

webm video:
Player failed to load "unityvideo://library/metadata/96/9630b4cd0c9e44345b951321160a53e6.resource"

Reproducible: 5.6.0b1, 5.6.0f1, 2017.01.a3

Note: Only reproducible with H264 codec