Anchors aren't the fixed point in virtual space we'd like them to be - they're points of known-to-be-higher stability (since you requested stability around that point by creating the anchor there) to center content around when you're close enough to it.

What you're observing with content getting less stable is known as the "lever arm effect". To address this, you need to place another anchor and reparent the transform there. The exact best thing to do with reparenting depends on use case and desired effect: for example, you might remove the old anchor and just keep making sure there's one closest to you, or blend between where different anchors report their spaces to be. Neither of these are guaranteed to be valid approaches though, they're just examples that might apply depending on use case and desired effect.