Some people could care less how they look and what others think of them. This is usually the deciding factor of taking off the visor or not. I would say it's rare to see someone riding a road bike with a visor on my rides. I'd guess the penalty of the additional draft is minimal, but what other things could be done to become more aero, such as clothing, aero position, weight, etc. If you're not too worried about all the other areas to become more aero, I wouldn't be too worried about the visor.