I'm not a tough guy. My first job was in a library.
For me, I don't ride because I'm tough, but getting out in all weather, doing anything -- cycling, skiing, walking, working -- makes me tougher, inasmuch as it re-fuels my brain, interrupts unproductive thought patterns, etc. It makes me more resilient to stress, helps me concentrate on hard problems, improves my confidence when dealing with people.
I'm not a strong or physically imposing person. I can move very heavy objects, using a long enough lever. A bike is a lever of sorts.