Reproduction steps:

1. Open the attached project
2. Open scene "scene"
3. Play
(use the GUI for the steps below)
4. Choose the amount of objects, spawn them and then destroy them
5. Notice the console output
6. Now select the flag "Create triggers?" and repeat step 4
7. Notice the editor freeze and console output

Expected result: Time destroying objects with colliders that are with and without "Is trigger" flag checked should be about the same.

Actual result: Destroying objects with trigger colliders takes significantly more time than destroying objects with simple colliders.

Test results: destroying 4000 trigger colliders takes 2100ms while destroying same amount of simple colliders takes 400ms

Reproduced on: 5.2.4f1, 5.3.3p2, 5.4.0b8