How to reproduce:
1. Open attached project "ADBV2SlowRefresh.zip"
2. Edit -> Project Settings -> Editor -> "Asset Pipeline Mode" -> "Version 2"
3. In Hierarchy window, right-click on Assets folder -> Refresh (observe the times it takes for the assets to refresh)
4. Edit -> Project Settings -> Editor -> "Asset Pipeline Mode" -> "Version 1"
5. Restart the Editor and the project
6. In Hierarchy window, right-click on Assets folder -> Refresh (observe the times it takes for the assets to refresh)
6. Compare the time it takes to refresh the assets (for more details you can check Editor.log)

Expected result: the time it takes for the assets to refresh on "Version 2" is similar or faster than "Version 1"
Actual result: "Version 2" is ~2 times slower

Reproducible with: 2019.2.10f1, 2019.3.0b9, 2020.1.0a11
Could not test with: 2017.4, 2018.4 due to Asset Pipeline V2 was not yet introduced