Steps to repro:

1. Open Unity;
2. Open attached project and 'test' scene;
3. Select 'Terrain' gameobject in the Hierarchy view;
4. Click 'Edit Textures...' -> Add Texture';
5. Choose any tiling texture (in my case 'Grass(Lawn)';
6. Add another tiling texture (in my case, 'Sand(Beach);
7. Select second added texture in 'Paint Texture' brush tool;
8. Select any brush, not having sharp edges (e.g. first left brush in the list);
9. Increase Opacity to maximum value (100);
10. Increase Target Strength level to maximum value (1);
11. Paint second texture on top of first one in the scene view (just a spot should be good);
12. Select first added texture (Grass) in 'Paint Texture' brush tool;
13. Decrease Target Strength and Opacity level (50 and 0.5 would be good);
14. Paint first texture on top of the recently painted second texture.

Actual result:
Sharp cut edges appear at the border of two overlapping textures painted on Terrain, in case if overlapping texture is the initial one with decreased opacity and target strength. Also second texture area (having greater opacity and target strength) significantly increases in size.
See attached video and actual.png for clarity.

Expected results:
- Blended overlapping textures should always have smooth borders;
- Texture with greater opacity and target strength should not increase in size when overpainting it.

Workaround is to paint blended texturs with brushes having sharp edges.

Notes:
- Reproducible in: 5.4.0b8, 5.3.3p2, 5.3.3f1;
- Reproducible on both OSX 10.11.3 and Windows 10;
- Not always reproducible for any tiled textures;
- Not reproduсible for brush having rough edges.