I long ago stopped using any sort of wearable bag when I ride. The lone exception being my lowepro AW200 case, for carrying my photography gear.

There is a reason the cycling gods gave us rear racks and panniers.