I never seem to get sick either as long as I am riding almost every day.

I had a couple of minor sniffle colds during the winter and would never use that as an excuse to cage up. Colds are actually less bothersome when you're exercising anyway as compared to sitting in a chair inside.