If it's cold and wet, gloves can get soaked and actually be colder than bare hands. Keeping your hands warm (in general) is about keeping your body core warm. No glove is going to help if your body is keep blood away from your hands in the first place (goes for feet too). Gloves are good for protection and for warmth against high heat transfer conditions, such as a sharp headwind or touching cold, metal things.

I don't always wear gloves when I ride. Pretty much, unless it's a race where I am using gloves for hand protection, if I can stand the conditions without gloves, I generally don't use gloves. If it's cold and dry, I use wool knitted gloves. I generally don't ride when it's wet and cold (<35F). Wet and kind of cold I'll generally ride bare hands.

Conversely, I'll generally wear cycling gloves when it's a long, hot ride to keep from getting blisters.