What hoodie do you have? I got the Chrome Cobra. Not to be a shill or anything, but it's pretty much perfect. Reach through rear pockets, perfect fit for riding (long sleeves, low back), perfect weight, soft, good zips. I love it so much. It was also cheaper than any other hipster-approved hoodie I found on the nets.

Edit: Dr., when I'm just riding-riding I wear a Merino jersey with Merino armwarmers. If I do have a destination, though, yeah I'd rather be looking normal. I think it's like the Merino hoodie keeps the outside temperature off me, then the Merino base keeps my inside temperature pretty well regulated, so the t-shirt doesn't really turn into a problem.