LSD,
The panniers shouldn't add that much resistance as they are some what in your wind 'shadow'. You should consider switching to a thinner smoother tire it will make a difference. What gear were you in when you were having so much trouble? If you are in a lower gear and peddaling at a higher cadence you can react and recover faster to sudden wind gusts. You don't need to wear lycra but should wear trim fitting clothing, a baggy jacket really adds a lot of surface area.

The NE winds have been a bear. I commute in from Oak Park and had it the face the whole way in. Just to get the benefit of a tailwind I rode up Damen to Hollywood and jumped on the lake path. It added 7 miles of riding but was worth the extra effort, it was like getting on the south bound express train.
Mark