That is the much bigger issue for me on riding longish distance in non-cycling clothes, I gotta be careful about which shirt so as to avoid the dreaded "nipple abrasion". Actually, sometimes I wear a monocolor jersey (wool, natch..) with the street shorts.

As for the shorts, I don't think I'd ever try much distance with a typical modern road saddle, I think most of those are designed to work with padded bike shorts. Actually, I started riding the Brooks stuff about 10-12 years ago, when I was pretty much a dedicated bike shorts guy. Did a lot of commuting, found that the street clothes were ok for commute distances, by process of discovery, found that increasingly longer rides were ok (with me, YMMV). So it was more by accident than conscious design, I just didn't feel the need to start buying more bike shorts as old ones wore out.