Here's some things for you to think about:

1. Heel strike. If you put panniers on a bike with fashionably short chainstays, you are likely to hit them with your heel.
2. Front-to-back balance. If you try to compensate for heel strike by moving your load rearward, your front wheel will be more lightly loaded than you're used to and your bike will handle funny.
3. Side-to-side balance. Loading a lot of weight on top of a rear rack will make starts more awkward.
4. How do you get onto your bike? If you pile a lot of stuff on top of a rear rack, you might hit it with your leg while getting on or off of your bike.
5. What do you want to carry back there anyway? I have a couple of rather big bags that I keep attached to my recumbents. Honestly, I'm not sure what's in either one at this moment. I do know that one spring when I opened the bag we kept on our tandem, I found a banana left over from the previous fall. The bigger the bag the more stuff you'll carry around with you all the time.