My laptop and my desktop both have only SSDs for storage, and every other folder, even those with far more files, show up rather quickly in File Explorer. My Downloads folder doesn't show anything but a progress bar for a few seconds. Just curious, is all.
I've had that issue when I had a shit ton of stuff in there. Clean it out, organize your stuff.
I also run into a situation with multiple hard drives in storage spaces when I haven't accessed the "drive" in a while, I have to wait for the platters to spin up and the heads to seek.