How much riding have you been doing? How many rides/week and what duration? And how long have you had that routine?

As stated above, if you're comfortable for 2+ hours, your fit can't be *horrible*.

While it might not be perfect, you can certainly help things by doing two things:
1) Try to increase your flexibility in your hamstrings. Tight hamstrings reduces your ability to rotate your pelvis forward, which will result in extra curvature of your lower back.
2) Increase your core strength. After a couple hours on the bike, if your core muscles are fatiguing, you'll slouch, again, putting extra curvature in your lower back.