My laptop and my desktop both have only SSDs for storage, and every other folder, even those with far more files, show up rather quickly in File Explorer. My Downloads folder doesn't show anything but a progress bar for a few seconds. Just curious, is all.
It's a special folder. It needs to resolve the location by reading the registry. Clean and compact your registry.