On the flats, weight isn't really an impediment to going fast. In fact the fastest TT'ers and sprinters tend to be big muscular guys. That's because on the flat, the relevant metric is power to frontal area. In the hills, it's power to weight.

I'd say keep riding, and throw in some interval work. Look at the sticky called BF Workout Cookbook at the top of the racing subforum. All sorts of ways to make yourself hurt in there.