I've never cycled whilst listening to music although I have considered it. Do I think it's more unsafe? Dunno, both sides of this argument I think have valid points but ultimately it's up to the individual.

One thing that occurs, if said cyclist is wearing earbuds or whatever and is slammed into by an idiot car driver who is totally at fault etc, one scenario I can see is the driver will most likely claim the cyclist was listening to music/had earbuds in and was distracted. (I'm making up a hypothetical but likely defence case). Now, the vast majority of the general public are not cyclists but likely are motorists and unfortunately there is plenty of cases where it's an us and them situation, especially in busy urban areas (London for example), motorists do not like cyclists. They see stuff reported in newspapers and online where lycra clad hooligans are weaving in and out of traffic, jumping red lights etc. These things do happen on occasion but then all cyclists get tarred with same brush by general public.
Now, it's absolutely the case you are not more likely to be hit by a car if wearing earbuds etc but I can see a situation where potentially the police, judge jury etc may consciously or unconsciously think "hmmm maybe he WAS distracted and wandered into path of car". You'd hope not but this is a scenario I can see occurring.

Just to be crystal clear, I am not taking sides here or advocating either way. We're all adults here so make your own decision. I'm just wondering if worse thing happened who would the police etc believe given there is a lot of prejudice against cyclists in some areas.

Here is just one random article about this :
Warning on headphones as cyclist ruled to have caused own death - Sticky Bottle - Sticky Bottle