First of all, you didn't acclimate in 4 days. Maybe enough so that you don't notice a difference under normal circumstances, but even if you were riding hard all 4 days - on day 5 you would be no where near full acclimation. It takes a couple of weeks of excercising to be acclimated to altitude, add to the fact that for most of that pass you were probably at 8-9k feet, where even I (acclimated to 4,800ft in Fort Collins) would notice a small difference I'm sure the heat was the issue.
That said, I'm from NJ. It gets very humid and hot there in the summer (not 100's, but 90s a lot) and its about as close to sea level as you can get without being on a beach. I too think its easier to ride in cooler, dryer climates, and for you the difference might have been big enough to make you feel more comfortable in Co. If you were to take Mt Indy and put it so you were down at sea level at the base, you would do better I think, but if it was super hot and sticky you personally might not adjust well to those conditions and it hurts your riding? I know I feel like a machine when I go down and ride at sea level. One time (the first time I went back to sea level after being in CO) I was pushing literally 35mph on flat ground (without sprinting, just pushing hard.) I was amazed I felt GREAT!!!!! Then I turned around for the return part of the out and back, and realized there was a solid 10mph sustained headwind.... LOL. The ride back sucked, and I had to be back by a certain time so I had to REALLY push to get back (and on top of that the wind picked up another 5-10mph sustained). Not a good day to judge performance LOL. The next day it was calm, and even though my legs were nuked from pushing into the wind without a wheel to suck I still felt better than and average day in CO as far as performance. This was in Virginia though, on the coast and it was about the same temp and humidity as Co.