People say that but it's not true, at least for me, once you ride some. When people start with deep wheels, wind gusts make you nervous. It's similar to getting bumped in a race. The instinctive response is to sudden tighten up which makes things worse.

If you just realize it's the wind and relax, it's nothing. Condider how big an aero profile a rider is on a bike, a slightly deeper wheel profile is nothing.