If I didn't have cycling clothes, I'd ride anyway. Being the not-quite-so-tender age of 40, I grew up riding in the late 70s and early 80s, when nobody I knew wore anything special for riding. I rode in gym shorts, I rode in jeans, I rode in whatever I happened to be wearing at the time, and I didn't think much about it. I also shifted my non-indexed 4-speed rear freewheel with levers on the stem, wrapped my bars with glorified ribbon, and had a padded saddle with "butt cheeks" molded into it (WAY too poor for a Brooks or other "good" saddle, even if I'd known when I was a kid that such a thing existed!)

Would I ride like that today? Sure, if we didn't have all the nice gear we have now, or if I didn't know any better. For my money, though, I'll be comfortable, and for me, comfortable means bike shorts and a jersey.

YMMV, as always.