The fitter you are, the faster/more easily you can ride without tapping into your glycogen, it isn't that you are storing more of it. Also fitter riders can more easily utilize fat for energy
Edit: maybe I'm wrong, *shrug*
Generally, this is close enough to the truth to be useful for the average rider, I'd expect. Did you check-out the graph? It's specifically about endurance sports like cycling... I should clarify that lots of your glycogen stores are also in your liver, and that you don't *only* synthesize glycogen at night when you sleep, BUT it takes you a minimum of 20 hours to replace your glycogen stores even with the best diet.
I also like the graph because it shows where/when/how much you use protein as well, and if you aren't taking in that small amount of protein after 4 hours of endurance activity your body will cannibalize your own muscles for what it needs... this is, obviously, counter-productive if you planed on using that muscle anytime soon
Bottom-line is that if you always hit a wall at 50 miles it's a combined matter of fitness, diet, and strategy as all have said already... I just wanted to toss-in the info about glycogen because when people talk about "hitting the wall" it's usually related to depletion of glycogen.