I always wear gloves. I also always pack a first aid kit with antibiotic packets.
My wife took a small spill while riding. She did not have gloves on. Despite cleaning the wound when we got home, she developed a severe infection from a relatively small scrape. It took a month to heal.
Besides providing a level of protection in an accident, they also prevent numbness from neuropathy that develops when one presses too hard against the bar while riding.