I take my backpack, whether walking or riding. When walking, I take out the pouch with tools. That leaves wallet, phone, map, papers, something to read, and any other things I need that day. In spring and fall I might bring a jacket, and in winter an extra insulating layer.
I love my backpack and it's always with me. I feel naked without it. I put it by bedroom door so I could grab it if there was a fire.
I have carried a backpack for at least 30 years and I've never had a backache. Maybe the pack serves as a counterweight to my belly.