How to reproduce:
1. Create and open a new Unity project
2. Open Package Manager (Notice it takes quite some time to refresh)
3. Close and reopen the project
4. Instead of waiting for the refresh to finish, close the Package Manager window
5. Reopen Package Manager (Notice it took less than a second to show the packages)

Expected result: The Package Manager loads quickly the first time
Actual result: The Package Manager takes some time to load the first time

Reproducible with: 2019.4.34f1, 2020.3.25f1, 2021.2.7f1, 2022.1.0b2

Note:
- On 2022.1.0b2 the initial refresh is fast (step 2) but after following the rest of the steps, it's slow to refresh after opening the project again
- On average, it takes about 9 seconds for the first refresh (depends on the project), but when closing and reopening the Package Manager it takes just 1 second