Here's my opinion, I'm gonna say it as simple as I can, take it as such:
Carbs are a fuel. Cycling burns fuel. Being alive burns fuel. The body prefers good fuel for good performance.
If I were you, I would keep riding as much as you are riding and adjust your diet to fit your needs. There is no point in lowering your mileage, really. It sounds like that would be punishing yourself, if anything.