I suppose it depends on your definition of 'tough'. To me a 'tough' guy drops me on a climb. Somebody that endures the cold better than me? He's just used to crappier weather. And he's not going to be riding me off his wheel.
You want to find out who's really 'tough'? Take the Illinois/NY/Mass cold weather ride weenies and the California legs of steel weenies and put them together for a 30 mile, rolling terrain with a stiff short climb or two, competitive ride. You'll find out who's got more starch in their legs - regardless of the temperatures.
OK, I ride in the wind, snow, rain, sleet, etc. But it sucks. And I don't think it makes me tough. I think it generally makes me pretty stupid/insane. Ask my wife.