I didn't notice it much under 14 mph average. Once I started getting into 14-19mph average ranges, draggy/flappy clothing was really noticeable.

I also live on a hill where its constantly windy and I regularly ride into strong head winds, so any aero advantage I can give myself helps quite a bit.