If I'm dressed correctly and have to wait for a ride to start for 10 minutes or something, I should be shivering uncontrollably. That's dressed correctly. It sounds like the OP is wearing cotton. Never wear cotton. None, like zero. I never wear wool either, except for my socks. Wear nothing but synthetic from the skin out. Never wear waterproof, even if it's raining, unless it's absolutely pouring and you don't ride fast. Of course you'll feel chilly if you stop for a while! If I have to stop and it's not raining, I take my jacket off to let the sweat dry.
The OP wants cheap. That's hard to do. Try Goodwill or some such thrift store. You'll need some sort of polypro undershirt and a pile shirt and a nylon windbreaker. Should be able to find those things. Head is a little harder. If it's really cold you need a balaclava or if it's warmer one of those thin Pearl Izumi skull caps under your helmet. Those aren't too expensive. Legs are harder to find cheap stuff for. You might be able to find some old wool ski pants. You'll need polypro longjohns under them if it's really cold. If it's raining and you can't afford proper rain tights like Pearl Izumi Amfib tights ($155), you'll just have to wear cheap rain pants with leg clips over the wool pants. Basically, if you're sweating a lot, wear thinner layers and nothing waterproof.