It's not possible to unbalance a bike. You'll automatically lean a bit the other way to compensate. I ride with an 8 pack of soda on one side of the bike along with my other crud often. I can't tell the difference.

I have ridden with one bag for 5 years now, no issues. Usually it's only 3 or 4 pounds of stuff, but sometimes lots more. This morning I had an 8 pack of soda and an old lead-acid UPS battery for the recycle bin in my bag.

Theoretically I suppose you could put enough stuff on one side of the bike to make it lean so far that the front wheel would want to caster (pull) towards the opposite side. I'd guess you'd have to have it leaning more than 5 or 10 degrees over, which I'd think given that the bag is within a few inches of the tire would mean that you'd have to put a heck of a lot of weight on it, like 80+ pounds. When I put 20 or so pounds on one side, I can barely tell that the bike is leaning.

This still wouldn't make the bike unrideable or even necessarily less controllable; it would just pull a bit to one side and you wouldn't be able to ride with no hands on the bar.