I have this issue, too, but it only happens to me with temps under 50/55. I wear a thin sweat-wicking headband that covers my ears. I've tried cotton before, but it feels odd and seems to muffle sounds more than I like. If the wind is high enough, I may even wear it when temps are over 60. I may look like a dork with short sleeves and a headband, but at least no ear aches.