I agree it's probably the engine more than anything else. Try doing a separate "training" ride of at least one hour (2 hours would be better) once a week with lactate threshold intervals spread throughout. The only way to get faster is to ride faster.

I'd also check to make sure your brake pads or fenders aren't rubbing, even slightly.