I use neoprene gloves for this very reason. I have 2 pairs; 1 for no liner, 1 larger pair for using with a liner. Days with stronger, colder wind, or wet, the liners make a big difference.