I haven't seen a photo of your bike, 12lbs puts it at the very lowest of the lows of road bikes in terms of weight. Which one is it BTW?

I'm sure you know that the UCI weight minimum is 6.8kg or 15 lbs (they are talking of lowering it) so the various stock bikes that are under that are pretty pricey.
Two things, why use such a light bike that is always going to have some issues with 20 lbs+ on it, and show us a photo of it with your panniers permanently attached to it.
Also, with your omm rack, how did you attach it to the seat post area or brake assembly? And on a 12 lb bike, aren't there concerns about the forces on a frame that is clearly not designed for having 20lbs hanging on the rear of it.

Pics or it didn't happen.....