I don't see why it would kill you, it just doesn't make any sense to carry the pack when you could have it on a rack. You don't even need a fancy rack and panniers thing, just enough rack so you can strap your pack to it. While I prefer panniers, a lot of them weigh quite a bit, so a golite approach of a pack and a cheap Al rack will work.
Around page 43 they go from backpacking to biking. Ray is the original golite promoter, and uses a really light approach that includes camping gear, and still under 10 pounds, but even he used a rack:
http://www.rayjardine.com/adventures...Bike/index.htm