RectMask2D masks a child game object even if it is in the boundaries. It only happens if the object (for example button) is rotated in a certain angles.

These angles are:

from 100 (or -260) to 178 (or -182) and
from 280 (or -80) to 358 (-2)

Basically in 2nd and 4th quadrants of the circle.

To reproduce:

1. Open attached repro project with unity 5.5.1f1 .
2. Open _main scene.
3. Enter play mode.
4. Slowly scroll the button through the top bound using mouse scroll.

Note: You should now see something like this: (Video: "case_882968_video.webm")
Results: Button disappeared before it exited the boundaries.

5. Scroll the button down and it appears again, but late as shown in the video.

Reproduces with: 5.3.6f1; 5.4.4f1; 5.5.1f1; 5.5.1p4; 5.6.0b9 .

The zip archive attached to this description contains repro projects for both 5.3.6f1 and 5.5.1f1 .