The only gloves I ever wear are full fingered ones when it's cold enough to want them. I live in a dry climate, so my hands don't really get very sweaty. I can wipe the occasional sweat from my face wit my jersey shoulder. I really hate padded gloves, and while there are unpadded ones out there, they can be hard to find.

The only drawback I see to this really is potentially falling. That said, I often do carry my full finger gloves on long rides because I'm likely going higher up where it's cooler. That lets me wear them for the decent, where a bad fall is mostly likely to happen anyway.

I suppose if I were to ever race again, I'd probably wear gloves for that. Otherwise, it's a risk I take for the comfort of not wearing gloves, I guess.