I also despise wearing a backpack while riding, in any circumstances.
These days, you can get all kinds of bikepacking bags, for people who have an aversion to racks. There are top-tube bags that go just behind the stem and just in front of the seat post, humungous handlebar bags and saddlebags, and of course custom frame packs. For the same weight as a rack, you can probably get two or three of those type of bags, and get all kinds of stuff in there. The frame packs are the most bang for your buck, as some small makers will build them starting at 100 dollars, with other features being extra. Two users on bikepacking.net's forums, 12wheels and jeremy11, are probably the cheapest way to get these, and are still well-regarded, while Revelate Designs and Porcelain Rocket are pricier but have a lot more users so there are more positive reviews out there.