My laptop and my desktop both have only SSDs for storage, and every other folder, even those with far more files, show up rather quickly in File Explorer. My Downloads folder doesn't show anything but a progress bar for a few seconds. Just curious, is all.
It's a special folder. It needs to resolve the location by reading the registry. Clean and compact your registry.
I've had that issue when I had a shit ton of stuff in there. Clean it out, organize your stuff.
I also run into a situation with multiple hard drives in storage spaces when I haven't accessed the "drive" in a while, I have to wait for the platters to spin up and the heads to seek.
That will be the NSA scanning your tentacle porn.