I sometimes listen to podcasts when I do extended rides, audio books might be a bit too much.

Headphones at reasonable volumes shouldn't get in the way of hearing traffic, but I still suggest you just get handlebar speaker bags. They can get plenty loud, double as noisemakers to alert peds/other riders in your route, and you'll probably have a much lesser issue than wearing headphones should you ever get in a collision.