Is there a specific amount of time usually considered "smart" after a race where you just allow your body a break?

In my case, I did a sprint tri on Saturday and everybody suggested no training whatsoever for at least 48 hours after, and as far out as 7 days.

So is there any sage advise to this, or is it all just anecdotal?

For Sprints, you could really go on a recovery ride the day after, and be fully recovered by the day after. Olympics are similar, just don't go for a run the day after, maybe two.
Half Ironmans are a bit of a different story, you probably have at least a week of easy to medium swimming and cycling, running can come a week to a week and a half afterward, but even then, not a fast long run.
I have heard the Ironman recovery period can be up to four weeks, but that depends on everyithing from your age, and what you ate, to your own personal physiologic variables.

Depends on the person. I know a guy who always rides the night of a race, helps him shake things out AND he trains and races a ton, so his body is used to it.

I personally don't take any days off unless it was an "A" race or I'm really beat up, then maybe a day or two.