The topic of heat gain is interesting.
Several years ago I noticed that heat didn't seem to bother me very much until I could hear the tar popping on black top roads. Once I heard that sound, it only took about 10 minutes until I was completely cooked and struggling to find someplace air conditioned or at least in the shade to sit.
Pretty much the same thing is true with me for wearing dark colored shirts. I have several black t-shirts and most of the time they don't bother me. Once I feel the heat building through them, however, they become unbearable in just a couple of minutes.