Like all things cycling related it depends on the individual. I'm comfortable in non-cycling clothes on any bike for any length of time I care to ride (12 hours a day). I can ride a road bike with drop-bars and spd-pedals in shorts, t-shirt and oddly enough I'm about as comfortable doing that wearing normal sneakers as I am wearing spd cycling shoes.
I ride in jeans all the time, but I'm sure to some people that would be torture.
Then again maybe it's not just things cycling related to which that applies. I'm sure some people wouldn't be comfortable wearing a suit and tie to work, and others wear that everyday and it doesn't bother them one bit.