If you don't track your exercise calories, then how do you know how much to eat and still keep your defecit managable on huge volume days? I mean, I ride a lot. Last week, for example I did 12,000kJ. Some days were less, and some days were more. Saturday was a race, and was nearly 3,000kJ. You have to account for that somehow.
What I do is try to establish regular routines for exercise and diet, and then try to tailor the diet for a calorie deficit. My point is that among the two, you get far more bang for your buck in paying attention to diet calories. The exercise calories can vary and it really won't make that big a deal -- it's almost not worth worrying about them as long as you're consistently getting exercise -- but diet must stay consistent. It is the area in which it is very easy to let a lot of them creep in unnoticed.
More to your point, I don't worry too much about calorie counts, as much as I do energy density (google 'volumetrics'). I make a diet plan based on a variety of foods with low density and then simply let hunger be my guide. If my physical activity for the week goes high, I find myself eating more frequently, but I stick to my plan so it's ok. I pay attention to the last 4 weeks of weight trend and make adjustments if necessary. By definition, if I find that I've lost 8lbs in the last 4 weeks, that I was running an average weekly deficit of 7000 cal. There's no need to micromanage it.
If you did 12,000kJ in one week, google converts that to 2,868kcal (correct me if I'm wrong here). You can easily order one meal at Chilis or your local rib joint that will go over 3,000kcal.