That's normal.
There are two things I discovered that help:
1) Eat more, start nibbling food right at the beginning of the ride and don't stop for more than about 15 minutes. If you can, have a meal at the 50 mile point just before you start feeling bad.
2) Do longer rides. Once you've done a few 125 mile, 150 mile, 200 mile rides ... that bad spot moves forward until it is out of the century range all together.