Wind- rain and cold are just a few of the problems you will have to put up with. I know you say you can't afford better clothing but look around in the charity shops for some windproof and rainproof clothing that can be better used for cycling.

Wind is a problem in that it will take away any warmth the body may have. Just think of wind chill. Rain will also cool things down a bit. Wind and rain together in the cold months and I will think about whether to ride or not. I have been out on aride in 40F temps and the wind took the temp down a bit and when the rain came I called it a day and went home. That was only after 10 miles but by that time I could not feel my hands or feet. Next week was at the LBS buying better gloves and overshoes for the Feet.

But I have done rides winds in excess of 50mph. As a headwind it was hard- as a sidewind it was dangerous.