A lot of those cycling-specific gloves are awfully cushiony. If you're running one layer of cloth tape on your bars down down chip seal roads, you'll be feeling it without gloves after only a few miles.

Mainly, though, I concur with the safety aspect. Knowing you can use your hands to protect yourself in a fall with relative safety to them might keep you from mashing up some other less fortunate body part as badly as you would have otherwise.