I have no problem with temps or Wind, what is my bigest concern is Snowy pavemnent because it makes me slow and my commute take too long.
Limits are a very personnal thing, When I'm not comfortable and confident on the bike I don't ride.