I once heard the following:

If the cold is in your chest, or you have a fever, don't ride.

If the cold is in your head, and no fever, then it's okay to ride moderately, since the cold is supposedly passing.

I imagine if your lungs were congested, then riding in cold air could make it worse. Same with a fever.