I don't listen to music when I'm on the road (and even that road is low use). However when I'm on a low use bike path or the like I'll listen but I don't use the head phones that plug into the ear and block out the surrounding sounds. I do want to be able to hear the sounds normally designed to get your attention but I feel not hearing the whispers is an acceptable risk versus my enjoyment of the music.
I generically don't wear the headphones when there are over a "certain amount" of congestion, especially if it is an area where kids are playing, lots of dogs are running, etc.