I'm late to this thread, but I met someone yesterday with an endurance problem that stumped me. I was halfway through a 70+ mile ride and hadn't taken a break yet, when I caught up with a guy my age out for about 15 miles on the MUP. We kept a good pace together on the flats, but on any kind of small hill, like the bridges mentioned in the OP, he couldn't keep the pace, and had to take a break within a few miles. He looked very fit, and talked about a lot of previous cycling experience, but he had no endurance anymore. It bothered him, and it sounded like it was a mystery, no diagnosed medical condition. I have no idea how he eats, drinks, sleeps, or what he does when he's off the bike, but I bet everything comes into play.
I don't even think about a break until I ride an hour or two. And racers scoff at that leisurely attitude. I think some of that is mental. I've done some long-distance hiking and bike touring and I learned a daily cycle of resting and eating, so I can ride or walk all day, and then get up and do it again tomorrow. Just getting to know your limits, and then staying within them, is a big hurdle.