Since you have single speed and don't need to shift, see if mittens would provide you with the dexterity you need - mittens are warmer than gloves since your fingers can keep each other warm. Along the same lines if you want a bit more dexterity but more warmth, try lobster claws, so each non-thumb finger has a mate.

I find that if I underdress the rest of me, my body diverts heat so my fingers get cold a lot faster. So make sure your torso is warm enough (without overheating of course - layers are best, and a jacket with pit zips) and make sure your ears/head are warm enough because the body will always prioritize head over fingers.

Still cold? Whether you go for mittens or gloves try a fleece layer on the inside, and a windproof shell on the outside. As you mentioned, its the wind that will get you!

Happy winter cycling!