Colder=slower, but it can still be fun!
I don't know which is most responsible for slowing me down: resistance from tights, wind resistance from bulkier clothing, more resistance from cold in mechanical parts (lube in wheel bearings or chains, or stiffer rubber in the tires). At some point it gets so cold you find yourself carefully choosing speed to get you warmer from pedaling harder vs. slower so the wind isn't so cold.