Opposite for me today - holy headwind batman! I have a 1x10 drivetrain and I was in 1 to 2 gears LOWER than average on my way to work this am!
I also hate the side gusts that take you by surprise - I have bar ends mounted inside my brake levers so I can get in to an aero tuck but you don't have QUITE as much control as when you're on the grips so when a gust hits your side, it's unnerving (I can send pics of my setup if this is confusing)