My weight has gone up and down most of my life. It *does* all come down to calories in/calories out, but what I've noticed is that the amount of exercise (as opposed to the intensity) can have a big effect. A few years ago when I was unemployed, worked in the yard most of the day (e.g., digging). It wasn't really intensive exercise, but I dropped about 15-20 pounds over a few months. When riding, I'll absolutely lose weight when doing long rides of 3+ hours on the weekends. If I do high-intensity 1 hour workouts, it doesn't seem to have as large an effect weight-wise.
Just my observation.