A jersey might seem a bit dorky, but no one would ever recognize you on the streets, or call you out for it, well, some might, but since you already wear a bib/cycling shorts, the looks part is unimportant. A T-shirt is great for laid back riding, but when you're putting in miles, a jersey cuts down wind and makes you more aerodynamic, having your T-shirt inflate like a flapping sail for two hours can be pretty irritating.