Because I'm a know-it-all.

Seriously, if you started the Tour with all your guys and somehow are down to only 4 riders because of crashes and etc, then no commissaire is going to not allow your 4th rider to count in the TTT. As I said before the TTT is early exactly for these reasons, so don't expect to ever see a team with less than 5 riders on that day.

Totally different from not having enough riders to start the whole tour.