When falling back to the non-directional shader we sampled incoming intensity from the lightmap but the shader interprets it as baked lighting. The indirect lighting contribution is stored in the other half of the lightmap so that is not visible. We would need a new shader variant to handle this correctly (and in a cheap way, llike just N.L)

Notes:
- Directional Specular mode will be “deprecated“ in 5.5.0f1 and will be removed entirely in 5.6. Currently, no fixes are planned for this feature.