That's quite technically illiterate. "Going fast enough" refers to the ground speed of a cyclist, but aerodynamic effects are actually determined by their airspeed. These two speeds are very different and a lot more unrelated that one might expect (unless you only ride in an enclosed velodrome). I ride against strong headwind quite often. And I do feel the aforementioned aerodynamic effects quite a bit, even though I'm definitely "not going fast enough". I'm sure everyone here knows very well what I'm talking about.

P.S. Personally, minority or not, I often ride at speeds that shatter car windows by sonic boom. KOMs on every segment. But that's not the point. The point is that one does not have to be that fast for the aerodynamic effects to matter (and matter a lot).