I also live in Virginia (the northern part). I'm shooting to ride almost year around this year. Unless there's serious snow or any ice, then I'm going to do my best to ride. I started this spring and the temps where in the low 30's. It wasn't nearly as bad as I thought it would be. I think if you try it a few times, you'll see that it isn't all that bad. Just get the right kind of gear, clothes-wise, because that makes it bearable. The arm and leg warmers totally make it do-able. I used to try it in just sweat pants and I was miserable... my knees usually froze up in no time. Trust me when I say, invest in some good winter cycling clothes and you might actually enjoy the fall, winter, early spring cycling.
Good luck
PBW