If you really want to lose the rack, maybe look at a saddle bag of some sort to take some/all of the weight off of your back.