First of all, you can find a plain new jersey at pricepoint for $20. You can find some pretty good shorts for around $30. Cycling socks, as phantomcow2 said, are not the best value. I found some tights for just under $40 and got a cheap Under Armour knockoff at Wal-Mart for maybe $15.

I've learned that anything that says "cycling" on it is almost always overpriced. You can find things like jackets, socks, under shirts, or even regular old shirts (if you don't want the pockets in the back, but that's what's great about cycling jerseys) elsewhere and they will work just fine. You can use old socks for leg warmers.