I keep the phone in a neoprene glasses case (found on the road side).

It stays in place in the jersey pocket & also keeps the phone dry.

I dislike scraping my hand on velcro pockets, & repeated washing may cause issues,

but hope it works for you.