Yeah, taking my jersey off with stuff in the pockets isn't usually an issue. Unzip, one arm at a time. If anything, my crap phone falls out. The thing is friggin indestructible anyway.
And as far as peeing, I know some dudes who opt to take off the jersey at a stop, but most of us just full down the front and whip it out...if you catch my drift.