I mostly wear Champion wicking fabric boxer/briefs with baggy shorts. Skin tight, great support for Johnson and the twins, and the fabric transfers moisture as well as any cycling apparel. I seldom need padded shorts -- I wear 'em only on really windy days on rural routes across open prairie to cut down a bit on wind resistance. My Lycra fabric covered saddles are comfortable enough for most rides up to 40-50 miles.
I suppose eventually the saddles will break down from acting as cycling shorts padding that can't be washed effectively. I just use some fabric freshener and laundry prewash in a spray bottle once in awhile. But Serfas Tailbones and similar Lycra covered foam saddles cost only $25 or so, cheap enough to replace every few years. Mine are more than 10 years old and still good. Really comfy, although they probably do cost a bit in pedaling efficiency -- I can feel a little bounciness on chipseal and rough roads.