Sounds like you're getting heelstrike on the panniers? Sounds familiar. Odd on a 20" bike.

One solution seems to be to ditch panniers and use a tall, skinny trunk rack. A taller rack helps. (I started a thread on just that fairly recently, and the folks here threw me some links that led to a few other links... and so on.)

You could also spend some time going through this guy's photos. He's got a bit of a frankenFoldingBike, but he's given a lot of thought to racks and adapters for mounting touring bags on a 16" folder. A lot of this stuff adapts well to commuting. (A handlebar bag and a small trunk bag makes for a dandy commuting rig all by themselves.)

On a 16" bike, you have to do all this stuff to even use a trunk rack, but on a 20" bike, I'm thinking it would let you comfortably use small panniers. Anyone here have experience with touring racks?