It looks to me like you're riding just enough to increase your appetite. If you can ride, say, 35 weeks out of the year, that's about 50 miles a week. You could get faster if most of that were high intensity intervals, but to lose weight I've found longer duration lower intensity is good for burning calories without making me ravenous. Maybe this winter you can set up a trainer and put in more time with a heart rate monitor. And use a calorie/exercise tracking phone app. If you burn more than you eat you WILL lose weight, but you have to find your own tipping point.

There are other considerations when it comes to building speed -- intervals, training blocks, recovery, nutrition timing -- but losing weight is low hanging fruit. And you'll feel better.