I usually find that riding with sore legs will determine how hard to push. If they loosen up after a few miles then I usually give it what I feel I can handle, but at times after intervals the day before, I will end up with a nice slow recovery ride but the flush out really was what I needed.
I found that sitting idle too long with day after sore muscles only prolongs the recovery.