The PolygonCollider2D being produced here is being added to the same Static Ground-Body (the same Box2D body) because no Rigidbody2D is being specified. The physics system doesn't know anything about GameObject. Not only that but each PolygonCollider2D isn't a single physics object, it's composed of 167 shapes each (see "PolygonCollider2D > Info > Shape Count" in the inspector). With 500 of them it results in 500 * 167 = 83500 shapes added to a single Box2D body.

Creating such things has been made fast (note the start-up time also includes decomposing the 1000 edge outline into 167 polygons!) however Box2D has a few areas of weakness such as when a LOT of shapes are added to a single body when it comes to destroying them. This is caused by the fact that all shapes (fixtures) are stored in a linked list.

I've also attached a modification to the script provided to demonstrate the speed-up by destroying the PolygonCollider2D in the reverse order they were created. This works because as you add a set of shapes (167 each per PolygonCollider2D here) they get added to the front of the linked list internal to Box2D. By destroying them in reverse order, Box2D doesn't have to search far to find the shape node. The slow part is the initial find which for each shape in this example becomes searching through 500 * 167 (approx) shape nodes inside Box2D. You can see the source code and culprit here: https://github.com/erincatto/box2d/blob/95f74a46c077825478f989df6a81003ca538c2c2/src/dynamics/b2_body.cpp#L232

We already destroy the PolygonCollider2D shapes in reverse order which is quick but the initial find is the slow part. By adding in a Static Rigidbody2D you are separating the shapes produced by each PolygonCollider2D into a smaller body linked-lists.

We've already replaced some parts of Box2D for significant speed-ups and we have plans for additional performance improvements that'll be added to the roadmap. This issue will be part of that by replacing the linked-list for a smarter look-up. Box2D suffers this issue in a few places including contacts and again, we'll be addressing those too.

However, those improvements won't be fixed against this bug report and whilst we do welcome the report, for now we'll have to close it as a 3rd party issue but it is something that will be addressed in the near future and isn't something that'll be ignored.