bobframe

I think that you can wear anything you want and have acceptable results.

Over my many years I have found that most sports (I'm most familiar with cycling and fly fishing) place unique physical demands on one and specialized equipment has evolved to meet participant's needs. For example, a short breathable raincoat with a serious hood and neoprene wrist closures has been developed for fly fisherman. It's probably too short in the waist to wear to the supermarket, but when you are wading in a river up to your waist, its nice to not soak the jacket.

Cycling is similar. Jerseys are a good example of a very design that is specialized to meet the needs of a cyclist and probably not much else. They wick moisture, have a long back to cover your lower back while hunched over, have a zippered front to allow one to ventilate, etc. Can you get by without one? Of course, but once you've ridden with a true cycling jersey and experienced how it's functionality supports you as you ride, you would be hard pressed to ride without one.

So, wear what makes you happy and what you can afford. But understand that some of this specialized gear really does work well for a cyclist and can turn your attention from your physical needs to enjoying the ride and the scenery. And at the end of the day, isn't that what it's all about?