I tended to get a shimmy with even a small handlebar bag so I went the trunk route and it helped. I got a rack slide and corresponding bag part from Topeak and made some mods to my Jandd rack and REI trunk bag so now I can keep my more valuable things like wallet, phone, and camera in the bag and just slide if off when I get to a store, etc.
You can see the Topeak QR piece here. I use a bungee over the clip to keep it on.


Here is the slide piece mounted to the rack.


If fits high enough to get at things when I'm riding. I chose this bag because it looks more aerodynamic even though it probably doesn't really matter.