I have 40 or 50 jerseys, and still ride in t-shirts mostly, with a fanny pack if I need to carry anything. Many of them are polyester and only get comfortable after i start sweating, which is never long when a person is bundled up in some overly-tight, gaudy, usually ill-fitting, polyester blend jersey that's covered in obscure ads, on a hot day.

I guess if you need to "look the part" then wear a jersey, but unless the ride lasts more than a couple hours, I don't use any cycling-specific clothing except clip in shoes, and maybe gloves, mainly to keep the sun off my hands.

I'm convinced that bike jerseys are more for conspicuous consumption and posing purposes, and to let the world know "I'm serious cyclist, DAMMIT!" than for anything that's really necessary as a practical matter for most rides I do.