Go. Most years it gets much colder than that in Michigan, and I've survived (thrived) through 3 winters on my bike so far. Besides, for only four miles, you can get through anything! For four miles, you could ride naked if you had to.
You do have lights don't you?