Considering that you're in reasonably good shape, I'd be more inclined to look at bike fit/riding technique.
Consider a professional fit session at a reputable shop. You might be surprised how much difference it makes. Alternatively, get an experienced rider to have a look at your position, foot placement, cleat placement, and similar niceties.
Are you "spinning" in lower gears, or "mashing" in higher gears? This can play a part as well.