I wear a short sleeved athletic T down to 50 degrees in the rain, and 48 when it's sunny. I see people bundled up when it's 60 and sunny, or wearing full length rain pants, a rain jacket, and a rain cover when it's rainy and 55 or 60, and I can't imagine how they're not an absolute sauna under their clothes. On the other hand, I wear long fingered gloves when it gets below 60, but my girlfriend keeps wearing her summer gloves down to about 40. It takes all kinds.

I think Bike Snob put it well: "there's nothing inherently wrong with cycling without a shirt, but I just find it wrong somehow." (paraphrasing)