I agree with the majority here: if you are not experiencing any discomfort, there is absolutely no need for you to wear gloves.

Personally I like to wear gloves when I ride for some of the reasons already mentioned. But I really doubt that the use (or non-use) of gloves has any bearing on nerve damage.