A couple of things worth considering:

1) There is no moral superiority/inferiority associated with your glove preference.

2) Accidents frequently happen very fast and in such a way as to make your intentions irrelevant.

3) The damage gloves could potentially save your hands from isn't catastrophic damage, but bad cuts and scrapes on your palms can leave you unable to use a hand or both hands for days or weeks (speaking from experience - nobody wants to have to have help to get dressed or use the bathroom).

But it's worth repeating #1. Wear them or don't wear them. Your call.