I ride 50mm wheels all the time, by the time it gets where the wheel become too much to handle, I won't be riding anyways because riding at my threshold at 12 mph isn't fun.

I've found my deep wheels are WAYYYY better than Mavic Ksyriums. The Mavics are horrible with crosswinds, its like the wind grabs and yanks them, while the deeper wheels get pushed gently by the wind.