Putting everything in a water bottle seems like it would work. No one really messes with those.

You can also try a fanny pack. I used one and didn't have problems. When riding I would turn it around so the pack part was against my lower back. Plus I would never forget to take it off my bike when I locked it up.