I do it on occasion and normally do not notice any issues, I just naturally lean a little bit to the non-pannier side to balance the weight. However, on my ride home last Friday, there was a good tailwind so I got up to about 25mph or so, and when the bike path turned a bit so that it was more of a crosswind from the non-pannier side (thus forcing me to lean further over), I did manage to almost lose the bike and had to slow down.