I've been blown over once, while waiting at a stop light with a load (around 60 pounds) of groceries on the back... just enough of a gust to knock me off balance, and the weight of the groceries did the rest.
That said, riding last weekend I left the panniers on the back of my bike... got blown off the trail twice and off the road once. Fun stuff. Next time I'll probably leave the panniers at home.
__________________
"There is no greater wonder than the way the face and character of a woman fit so perfectly in a man's mind, and stay there, and he could never tell you why. It just seems it was the thing he most wanted." Robert Louis Stevenson