Id ride without a helmet, but never a mirror.
The most important safety device there is. If
it is a 'distraction' as the detractors claim, you have
the wrong type for your riding and bike set-up.