To reproduce:

1. Open attached project
2. Select my_vp8_clip_with_alpha video
3. Enable transcode
4. Select h264 codec
5. Apply changes

Actual behaviors on Windows:
Video fails to transcode and following error appears in console:

WindowsVideoMedia error 0xc00d36b4 while reading C:/Users/audrius/Documents/VideoTest/Temp/importedVideoClip.m4v

Context: IMFSinkWriter::SetInputMediaType (audio)
Error details: The data specified for the media type is invalid, inconsistent, or not supported by this object.
Track types:

UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

Actual behaviour on macOS:
Transcoding is done without any issues, but if Keep alpha is selected transcode doesn't start and following warning appears:

Warning: Selected transcoding size (8192x2048 = 16777216) exceeds H.264 profile 5.2 limit of 9437184

Reproducible: 5.6.0p3, 2017.01b2