I've been riding bikes my whole life and never once worn cycling shorts with any sort of padding in them. I've gone on some fairly long rides too and ride regularly.
In the past, I had some sort of gel saddle that was quite comfortable for many years and have since passed it on down to a friend though can't remember the brand. It was one of those that was probably more marketed towards commuters and the like but it worked great for me on multi hour rides.
I had a Brooks B17 for a while but just could never get comfortable on it.
These days, everything I own has some sort of mountain bike saddle. My main road bike and my commuter have similar, slightly older Bontrager mtb saddles. Both of those are excellent. Not necessarily light weight but nowhere near as heavy as your standard comfort saddle. Good balance of soft but supportive.
My lightweight carbon road bike has a high end WTB mtb saddle with titanium rails, quite light. A bit on the firmer side than the Bontragers not too bad. I took it on a 3 and a half hour ride yesterday and my butt was only very slightly sore by the end. Granted, I don't ride that particular bike all that often, if I did, probably wouldn't be an issue at all and it's already minor to begin with.
My mountain bike has a newer-ish Specialized mtb Body Geometry and, as others have mentioned, is quite comfortable. I have another bike with an older Body Geometry saddle and that thing is quite the couch cushion, I really like that one.
For whatever it's worth, the leg wear I use while riding is all over the place. On the commuter, it's typically rolled up Carhartt pants. On the road bike, it's usually a pair of casual fitting, non-padded cycling shorts (one could call them lifestyle shorts I guess) with conveniently zippered pockets. The kind you could wear around town and look like you're wearing regular shorts. On the mountain bike, it's anything goes. I've worn my Carhartts, the previously mentioned cycling shorts, board shorts, whatever. All have served me well.