I have ridden with an iPod and only on long isolated solo rides, but it is very rare, haven't yet this season (since Jan). I actually have a one eared pod phone for the purpose.

As to being distracted: what I really don't get is with all the wind noise, I can't hear anything behind me anyway. And unless the iPod is loud, it is difficult to hear it over the wind noise too. I suppose if you turn it up just loud enough to hear over the background noises, than you should be okay, and it shouldn't be an impediment to "hearing" safety. But every rider has his own comfort zone and opinion on using one.