Reproduction steps:
1) Open user's attached project
2) Open "TestScene"
3) Select "PrefabInstanceWithColliderReplaced" prefab in hierarchy to see components' (especially colliders) behavior
4) Start scene

Prefab which bugs reproductions steps:
1) Create a new game object.
2) Add a BoxCollider2D.
3) Add the ScriptRequiringCollider2D component I have attached to my bug
report.
4) Create a prefab out of this game object.
5) Instantiate another copy of this prefab into the scene.
6) Add another collider that will satisfy the component dependency, say,
PolygonCollider2D.
7) Remove the original BoxCollider2D from the prefab instance.
8) Play.

Note:
The more you starting scene and ending, more errors and colliders you get.

Note 2:
Works only with users attached prefab (watch attached gif).

Note 3:
Bug appears on both 3D colliders and 2D.

Expected result:
If component exist on object, shouldn't create a copy of it.

Actual result:
Creates as many copies of component as many times scene has been started.
Also sends error on start and stop scene:
"Error removing component when merging prefab changes: Can't remove BoxCollider2D because ScriptRequiringCollider2D (Script) depends on it"

Reproduced with:
5.0.4f1, 5.1.4f1, 5.2.4f1, 5.3.0f4, 5.3.3p2, 5.3.3p3, 5.3.4f1, 5.4.0b8, 5.4.0b10