I think people only perceive it that way because it's "unusual". But like any sport, if you have the right equipment and gear you're all set.
The two comments I get the most are "aren't you cold/freezing?" and "don't you slide, spin out or fall a lot because of snow or ice?"
No / I'm usually warm enough to so warm I'm shedding layers. Only rarely am I cold -usually a miscalculation about conditions.
No, never had any issues with slips or falls due to traction. I dread riding on wet fall leaves on my road bike far more than ice/snow, even black ice.
It's all in the preparation and gear