I stopped wearing my gloves after I started to develop a tan line that made my hands look like ghost compared to my arms.

I never leave without them anymore though. Even just for a rare case where your chain gets stuck, you are going to be thankful you have gloves then. I don't use padded gloves so they wouldn't effect pain but they increase grip and the rides just feel nicer with them.

If you notice hand pain you might want to look into better grips as well - check out ergon grips - makes a world of difference for hand pain - glove or no.