I was thinking this before and as the Tour was starting...
my guess is that him having to lose weight so late in the spring probably hurt his ability to properly train and peak for this race. And it also probably made him more succeptable to illness. Maybe he got a little sick in the cold early in the race. Or even without illness, maybe the cold simply hurt his overall form (some bodies tolerate it better than others). And if he arrived a little overweight, the weather probably did kill his form (that is what happened to Indurain, or so I've read).
Compound any possibilites with the fact that La Mongie is really steep... something that would suit a more natural climber better than Ullrich.
And then it could also just be the competition simply getting better. I think Armstrong may be on the best form of his career, and he wasn't able to ride off from Basso so far, for example. IMO, teams with DS's like Bruyneel and Riis are probably more professional and advanced in training methods, while the methods of other DS's might be a bit behind...
I'm just playing guessing games on all of this.