That's one of the reasons why I don't use panniers. My main backpack goes into a Nashbar Townie*. Rain gear is kept in my Banjo Brothers backpack. I only use the Banjo Brothers if it is cold and is or will be wet, otherwise I don't carry the rain gear with me. And all of the tools are kept on the bike.

*Allows me to look like a normal person off of the bike, and I can always put extra stuff in the basket.