Some of these have already been mentioned...
I got my rain gear at a thrift store. I paid $3 each for the jacket and pants. I also use a $2 shower cap over my helmet. And yes, 2.5 gallon zip-lock bags, over your socks will keep the water out. Just be sure to tuck the top of the bags inside your rain pants. I only do this if it is cold. I enjoy riding in warm-weather rain, without the rain gear.
The rear fender, pedals and seat on my shopping bike came out of my LBS dumpster.
I built the rear rack, for my commuter out of aluminum angles from Lowes. I think it cost me about $7.
If it is cool, (i.e. - not cold), tear a sleeve off an old t-shirt and use it as a skull cap, under your helmet. The top is open, but it keeps the wind off your ears and forehead.
After hunting season, Wally-world puts most of their hunting clothes on clearance. You can get wind/water proof gloves for $3 or less. Just as long as you don't mind riding around in camo or blaze orange.