Fit might be an issue, but it also could be just plain old muscle weakness (and I say that politely). You perhaps should look at doing some crunches and dorsal raises to help improve your core strength. You are generally riding in one position for much of the ride, so your back muscles (and the main support ones across the stomach) are likely to fatigue and get sore.
I know that having come back after a year-plus layoff from distance riding that the left side of my back has presented some soreness on the first populaires, but then has improved on the 200 and 300.