Bad Advice! "Fully supported" does not mean they carry you when you're tired. We don't know many specifics in this case. What is "slow?" Normal advice is to go 2-3 mph slower on the tour than you would on your training rides. Would that put you behind the suggested advance lines, i.e. behind the SAG vehicles?

I suspect nutrition or hydration. Try eating an energy bar and downing a bottle of water immediately after your rides and see how that helps with recovery. Longer rides can deplete your body's stores and if you don't get nutrition in the first 20 minutes after a ride, your body will start cannibalizing muscle tissue. Weight loss and training aren't necessarily compatible activities. Obviously if you don't think you'll be able to make it you should bail while you still can.