I used to wear cycling shorts for every ride. Lately however I've moved to very lightweight underwear and nylon cargo pants.
I feel more comfortable in them. When I get off the bike, I can usually just blend in with the rest of the world.
With bike shorts and jerseys, the slightest hint of a paunch is exaggerated to make the wearer look awful.