Agree on the protein after riding bit. It sounds like your rides are a few days apart. If so, you could try this: try a short ride every day, even if you feel tired. Don't go hard, just pedal along, barely breaking a light sweat. Say 30-45 minutes. On a trainer is fine. Then every few days, do your harder long ride like you are doing now. This will take 2-3 weeks to take effect, but I think you'll notice a big difference.