I had something similar for my Crater Lake trip. Full panniers (like sails on the back of my bike). It was generally just as I'd crest a hill that I'd get the full force of the cross-wind. Then it was ok once I dropped down in elevation again.

Just had to be careful every time one reached unprotected open space at the top of a hill.