Looking into this with the memory profiler it doesn't seem like this is a leak. I simplified to just use buffers with no GCHandles and the same behavior is there.
It's reserving managed memory, but empty. If I run an import the first import takes a second, but the second file is near instantaneous. It reuses that reserved memory space.

It makes sense why you'd want that model for importers. If I'm going to import 100 files of roughly the same size, I want it to reserve the memory once and then reuse it.

I looked into it for a day and a half and there doesn't appear to be a way to compact that memory space. There are longer term plans to make this architecture work better, but we don't really have a simple fix.

The recommended way to deal with this is to use buffered file streams and object pooling to control the amount of memory being used. I talked to Alexandre and he said they have a solution for now.