I have never worn a cycling jersey, when it's warm I just wear a t-shirt. Maybe I should be terribly worried about the 0.1 mph that a loose t-shirt slows me down as compared to a jersey, but I just like being comfortable when I ride, and regular street clothes are comfortable for me. I don't really get hemmed up about what other people wear when they ride. Wear whatever is comfortable for you.