A jersey can be nice because it should wick sweat better than most other tops, and should fit snugly so it doesn't flap in the breeze.

I've had shorts where the pad hangs kind of loose when I'm off the bike, but are perfectly fine on the bike. You definitely don't want the shorts chafing when you're riding. In terms of fit, I prefer my bike togs to be painted on, but that's just me.

There are cyclists who wear regular streetclothes, even for ultra-distance events. I'm not one of them.