If this already came up I apologize for missing it...
It sounds like you're a 'weekend warrior', yes? (no offence, just clarifying). If you're average max rides per week is equal to or less than 1 this might be a big part of the answer to your question. Some of the other advice is great, i.e. take in a small amount of protein, and the proper calories in complex carbs as you go (such as with Hammer Perpetuem and a good gel) but that only helps so much... Much of your energy comes from the glycogen stores in your muscles -- the more you train the more you store... if you ride all the time you're going to have a huge warehouse of glycogen but if you only ride on the weekend your body will never decide that it needs more than a quickie-mart worth of inventory, if you will. At least that's my experience....