My roommate wears his rugby jerseys when he rides. Only thing is they ride up in the back, which cycling jerseys are designed to prevent. I just wear sleeveless t-shirts or a cycling jersey. I live in FL, so either I wear the jersey or deal with being wet 24/7. The only alternative would be to buy some sort of athletic wear, which I wouldn't be able to justify because I could get a jersey designed for cycling for the same price.

As far as shorts go, I usually wear... shorts.