So ride less and make sure you enjoy it. When I started out, I soon ended up at about 120 miles per week and 10,000+ vertical feet per week. I eventually cut way back, but I absolutely love my rides. Sure, I'm **** (stick in your favorite term) slow, but I really enjoy my rides. I can complete the rides I want to complete, and I'm not training for anything, so why should I ride when I don't feel like it? If I'm off the bike enough, then I always get around to looking forward to some crazy mountain rides.
My riding totals have been pathetic this year (I've been on the bike maybe 20 times total), but I went out and enjoyed a 60 mile ride with 8000 vertical feet of fun descents last Sunday, building up to one of my favorite rides: the Pie Ride, in its 100+ mile, 10,000+ vertical feet of descents glory (and with a wonderful pie stop). I'll probably be the slowest in the group when I do the Pie Ride, but that's okay. I'll just plod along and enjoy the scenery, have fun on the descents, and eat lots of pie.