in my experience - and based on google searches - if you have a fever, you should rest. if not, it's basically up to you.

if i just have a cold, i almost always ride and i almost always feel better afterward. on one or two occasions, i'm convinced that i actually got over colds faster because i exercised (obviously not scientific, but so what). i don't usually ride hard; maybe 1-2 hours at a pretty moderate pace. but i listen to my body. like you said, if it "sounds good," i'll do it, but if i'm really feeling worn down and crappy, i skip it.