Yup, that one disadvantage is truly there. That said, I commute on a single-speed, and the usual nightly 10-mile ride home directly into the west wind only makes me stronger when I am out on my "real" bike.

I personally don't mind a backpack in the heat, but in the winter, it is trunk rack or panniers all the way. I actually just use them year-round now, but it wouldn't be a big deal to use the backpack in warm weather for me. I just hate wearing a backpack over a jacket and base-layers when it is freezing or below. I really, really hate it.