I have a cycling wallet kind of thing that holds some money, copy of ID, keys, Chapstick, and my phone. And the whole thing goes in my jersey pocket.
If I'm commuting, the phone's in my backpack.