Steps to reproduce:
1. Open attached project
2. Enter Play Mode
3. Move either of the three objects (cube, capsule, sphere) aligned with the black ray in front of the ray's source

Actual result: the ray hits whichever object that is closest to it (hitrayissue.mp4)
Expected result: the ray hits only the object in a corresponding Layer

Note 1: the script being used is Raycaster.
Note 2: the layer mask is set via a public variable in the script using the Inspector
Note 3: all three objects are in different layers, the one used as a Layer mask in the attached project is for Layer3

Reproducible on: 5.6.1f1, 2017.2.0f3 , 2018.1.0a3