Sounds like you're not eating enough in the first few hours and are having a minibonk. Machka's prescription should be very helpful.

Are you starting the ride with faster people and then end up getting dropped? It's very tempting to go with the faster people, but you have to be very careful not to blow through all the glycogen and end up with leaden legs. I've spent many hours regretting hanging on ten minutes too long with the faster riders.

I do not think kk4df is correct that we uniformly use up all the glycogen and then switch to blood glucose at around fifty miles. Maybe he/she can provide a cite for this?

If you keep fueling up at 250 to 300 calories per hour, and you are careful to "avoid unnecessary accelerations" (as Jan Heine says) then it is my understanding that you can avoid using up your muscle glycogen. Instead, at randonneuring speeds, we are getting about half our calories from carbohydrates eaten at 250 to 300 calories per hour, and half from burning on-board fat -- at least, that is what the Hammer people claim, and is the same as in the book I'm reading called Paleo Diet for Athletes, by Professor Loren Cordain and Joe Friel. The quote from the book is "Fat burns in a carbohydrate fire", which is why it is so important to keep eating carbs as you ride.

Anyway, certainly it has been my experience that fueling properly and riding at a steady, hard pace does not empty your legs of muscle glycogen. If it did, then there are quite a few very hard sprints that I've done at the end of a 200K or on the second or third day of a 1200Km that I don't think would be possible with "empty" legs.