Actually you are being tough and slow rather than tough and fast. When the big European road races go over roads even fractionally that bad, the riders switch to specially built racing frames that can take wider tyres, or just cross frames, because 23s suck energy on poor road surfaces - they can't flex enough to absorb road variation, so pedaling energy is wasted. And pro riders only use 23s in the first place because they are moving at speeds where the slight aero reduction is worth the increase 23s bring in rolling resistance - for a sports even more like commuting, randoneuring, 28s and 32s have been proven faster.