I carry a Camelbak on tour. I like having the extra water capacity (on top of my 3 water bottles - I don't have them all full very often, but I like having the capacity if needed.) I also like how it keeps cold water cold for much longer than bottles on cages.

In college I rode my bike the 4 miles to school every day and carried all my books and things in a small backpack.

Both of those comments illustrate my feeling that a moderate load in a backpack isn't all that bad, if that's what you want to do.

However, once, when I was really young and stupid, I carried an exterior frame backpacking backpack on a bike for about 10 miles. Boy, was that dumb! It was totally unwieldy and the frame hit my head.

If you want to carry a moderate load on a backpack it should be fine. If you need more, consider combining a backpack with a small rack-mounted load.