I'm somewhat the opposite and find that riding seems to improve recovery. Sometimes I wonder if riding in miserable weather while sick is so hard on the body that it makes the germs seek out a more gracious host. But overall, my experience is stress and activity, even though it's difficult works in my favor.

OTOH- don't wear decent clothes when riding with a head cold, especially if it's cool out. It's a messy affair.