I'm car-free, and ride the bus when the conditions are too nasty to ride to work. So I'm on the bus less than 8 weeks per year.
I get a cold/sniffles/congestion about mid-October, and more or less keep it until mid-March, and ride through it; the exercise promotes "productive expectoration", meaning I can spit it out and get rid of it. So a cold doesn't even faze me.
Fever, yes; generally, when I get a fever, I get a little vertigo in the bargain. But my last fever was... '05, maybe?
You could always do the "nipple test"; if the discomfort is above the nipples, ride; if it's below, drive or stay home.