You should always track your diet and see if you are eating pointless calories, but otherwise I wouldn't do anything too drastic. There are all kinds of fad diets, I'm not a fan. I find that I need to stay on my diet even when I'm riding a lot or else I gain weight. The human body is a greedy thing when it comes to food.
I am a worrier, I just don't let that stop me