I do a pannier or nothing (I keep about 4 clothes changes plus toiletries, towel, etc. at work), but I don't mind using my Camelbak Trailblazer, in fact I used it exclusively for the first 6 months of 2008. Even though it's supposed to be a light hiking pack, not a bike pack, it's quite comfortable and if not stuffed to the gills, holds the weight down low.
Also, the "Air Director" back panel keeps *most* of your back pretty cool.