To reproduce:
1. Open the attached project and the 'LayerMaskBug' scene.
2. Review the 'LayerMaskCheck' script. Note that script feeds the value of the layer mask field to LayerMask.LayerToName(int) and outputs the result to the console.
3. Play the scene.
4. Select '__RaycastTester__' in the hierarchy view.
5. Note that the 'Mask' field, which is a LayerMask field, shows 'Default'.
6. Note that the console message is 'TransparentFX'. (Bug)
7. Change the value of the 'Mask' field to 'TransparentFX'.
8. Note that the console message is 'Ignore Raycast'. (Bug)

To see the bug even more clearly, paste this line of code after line 38:
Debug.Log (msg + LayerMask.GetMask (LayerMask.LayerToName (mask)) + " (" + mask + ")");