I only wear earbuds in traffic. The surrounding noise drowns out most of my music anyway. And along with my music that noise drowns anything else out, like other cyclists. There's really no situation where an important noise that would be covered by my music wouldn't already be covered by *everything* else.
I take them off on MUPs and isolated roads. Things are quieter there and the important sounds (ding ding, "on your left", really, really distant cars) are more easily drowned out.