Sometimes a minor change in handlebar/stem height, saddle height and/or fore/aft positioning can solve comfort issues.

Since switching my Univega Via Carisma from a riser bar slightly above saddle height to an albatross bar at or below saddle height, I've swapped saddles around a few times trying to find the right setup. Pretty much anything is fine for 10 miles or so, but on 30-50 mile days I notice some discomfort.

I went from a heavily padded comfort saddle -- Lycra fabric over foam -- to a thinly padded old Selle Italia road bike saddle, to a more padded Selle Italia Q-Bik, and back to the thinly padded older Selle Italia. The older saddle is a split configuration with a flexible shell and quite comfortable with the right setup.

I've found that some minor adjustments, 1/8" here, 1/4" there, improved saddle comfort. No problems on my last 40 mile ride, mostly gravel.

So your saddle may be fine. And you may not need to toughen up any part of your body. Maybe a few tweaks here and there will do the trick.