The team holding the yellow jersey is expected to control the pace of the peleton (assuming they want to keep it), which means putting their riders on the front and breaking wind for (and on) everyone else. It's hard on a team to do that for three weeks.