I'm in the habit of checking my resting heart rate first thing in the morning and that indicates to me when I need a rest day (elevated, and really sore legs notify me, too).
I almost never schedule a rest day, but my life always puts them in. I've missed a month of riding so far this year (professional conference away, Mardi Gras, and a couple of post-surgical weeks). They probably don't hurt, but:
I'm trying to learn to accept them...