No, you're not alone at all. I have found that as I have aged, the cold bothers me a lot more than it used to. This, despite having a good jacket, tights, ear protection, etc. The wind/cold you speak of is very intense here in eastern MA as well. Yesterday was just plain nasty, with a very intense wind from the east. And, if it's not from the east, it's from the west. The west wind is usually quite strong. I call it "plastic trash can in the street weather". I think that's self explanatory. Sometimes, the west wind is warm, and sometimes it's just as nasty cold as the east. Go figure.
Summer heat and humidity? Bring it on! Nothing pleases me more than a ride on a warm summer evening.
This is why I'm looking at warm-climate areas for retirement. I know, me and about a gazillion others.