I sometimes listen while road riding. You have to decide for yourself based on how and where you ride and what you believe the tradeoffs to be.

I have yet to find an in-the-ear headset that doesn't make wind noise worse.