Even if the saddle doesn’t tear up the material, it will still damage the seat of your pants with a saddle shaped faded worn spot. Cotton just won’t hold up (long term) to the effects of cycling. Tech fabrics like nylon, poly, etc will hold up much better but with constant use eventually wear out too. I can appreciate your desire to simply hop on and ride, but if it’s more than an occasional thing you gotta know your pants will take the toll.
In my younger days I rode all the time in regular street clothes but it’s expensive to constantly have to buy new pants when the butt’s worn out. Slick saddles only help by delaying the inevitable, not ending it.
-Kedosto