I prefer a backpack until it gets to be above 60, and then I use a pannier to keep me cooler. The backpack feels more stable and comfortable to me, and it sounds like yours doesn't fit you too well. I hate riding into the wind, and the backpack reduces the effect of the wind while the pannier feels like some sort of parachute slowing me down.