I would start 30-minutes into a ride if I'm doing a 3-4 hour ride. There's a limitation on how much your body can absorb per hour, about 200-300 calories max. So if you're going to be burning off 4000 calories and have only 2000 stored, you're going to need to ingest the difference. In 4 hours if you start eating right at the start of a ride, you're still only going to get in 1200 calories max. That 800 calorie difference can be from fat-metabolism if you're fit enough and ride slow enough.
"My logic was that I should have enough stored up to last the first 2 hours, but I'm experimenting with starting sooner to be more in line with the idea of drinking before you are thirsty, but with food. I normally save the energy gels for either a tough section or when I really feel drained."
The problem is you can't absorb 2000 calories in the remaining 2 hours of your ride. You'll only get in 200 calories by the time you bonk at 3 hours. Then it's creeping and crawling home that last 25 km at 10km/hr while you desperately scan the ground for scraps of carbs to eat...