I've ridden with a cold before, but it took the enjoyment out of it (which is part of my reasons for riding).

One rule of thumb that runners follow is that it's OK to exercise with a head cold, but not with a chest cold...but, as mentioned above, don't overdo it...you can wear yourself down and make yourself sicker (more sick?) and then you'll have to lay off that much longer. Listen to your body, or ignore it to your own peril, as the ex used to say..