You definately need to wear clothes when its cold. Clothes are what you need.
Maybe you already have clothes you could use; most people have some clothes. Try raiding your current sports wardrobe before splashing out. You dont have to get expensive cycling stuff. I find that general outdoor/sports stuff works fine and no-name models work about as well as major brands.
Essentials are a breathable windproof shell. These can be picked up cheap and the more expensive waterproof ones will get clammy. Some tights or sports pants for your legs.
For an inner layer you can use your cycling jersey. A full-arm wicking T shirt is a good investment.
For midlayers , thin woollen sweaters work fine.
For extremities, gloves, woolly socks , neckwarmer tube, beanie hat.
If you have to stop you need an extra layer so carry a spare jersey/woolly jumper, hat ot whatever to stop you chilling out.
Get some clothes!