I guess it depends on the cause of built up fatigue as you say. If it's a considerable amount, take the two. Or, maybe do some off the bike easy exercise. Easy swimming maybe?

Having said that, I find rest days the most challenging. Housework is just not as fulfilling.