I dunno, if you're trying to train to be the ultimate spandex monkey then maybe a day off will help gain the speed to hold the front of the peleton on your next crit (or however those racers refer to it) but if riding your bike makes you happy and you'll be happier if you go for a ride, then by all means, throw on the spandex and the clunky shoes and shuffle yourself out the door.
My body is currently telling me that I can't continue to run 40 miles per week unless I figure out a way to consume more calories. I can't figure out how to eat more and still eat healthy though, my stomach is pissed about me cramming it full constantly, it wants to feel hungry occasionally. I really hate it when my serious running starts to conflict with mt recrational riding. If one of them is going to have to take a break, it isn't going to be the bike.