Why destroy them immediately after tho?
They didn't destroy all of the buildings, only most of them. Real estate usually only increases in value while most people value historic preservation right up there with stubbing their toe if they even consider it. Time and profit kill the wonderful buildings. I've visited the old fair site in Queens a couple times on field trips. It was interesting. Apparently, Olympic games don't always cover their costs..
Q: How could they build these buildings so fast in the era of horse and buggy?
A: They hired a lot of people with a lot of hammers. This was a time that preceded income tax. The rich were rich in a way that Elon Musk can only dream of and a bachelor could get by working only two days a week.
https://en.wikipedia.org/wiki/History_of_taxation_in_the_United_States