For me, the convenience in not having to swap tires is worth it. Also, the gearing I want on fast road tires is often different from the gearing I'd use with wider off road tires, given different terrain being ridden. So two sets of wheels also offers two sets of gearing, without the need to swap cassettes.
__________________
You could fall off a cliff and die.
You could get lost and die.
You could hit a tree and die.
OR YOU COULD STAY HOME AND FALL OFF THE COUCH AND DIE.