I tend to prefer t-shirts over jerseys, although when I do wear a jersey, it's just a solid color... I don't like to pay $50 or more to be a rolling advertisement.
Bike shorts might look dorky, but they do make the ride a heck of a lot more comfortable.
One time I was short cutting through a local high school parking lot, and some kids were out there. The guys started making fun of me and yelling at me about my shorts. But I noticed that the girls weren't yelling and making fun of them. ;-)