I think it would help to log all your food at least for a week or two, so that you know where your calories come from. Check out
www.loseit.com. And yes, it means that you have to get a food scale and weigh everything that didn't come pre-packaged.
Do make sure to eat a lot of veggies and greens. You didn't mention alcohol. Do you drink alcohol? If you do, you have to account for that as well, there's a lot of calories in beer and wine.
How often do you have long rides (2 hours or more at once)? Is it just on weekends? In that case, you can load up on carbs the day before the ride and the morning of the ride, eat a hearty balanced dinner after the ride, and try to maintain a restricted diet, shooting for 1500 calories/day, the rest of the week. If you can stick to that plan, you'd be losing 10 pounds per month, which is at the high end of what's considered safe.