I'm not sure of your age but I'm 40 and have only been riding for 7 or 8 months. I mostly ride by myself and try to do at least 100 miles per week. Living here in Central Alabama I'm not far from where you are and your description of the rolling hills sounds about what I ride each day.
About 2 months ago I decided I was ready to go for a ride with someone and my step brother is a marathon runner and has a road bike that he bought so he could keep his fitness level while recovering from a running injury. We met up one day and he took me on a route he had ridden with a group of guys where he lives.
Well, lets just say I was sucking wind all day and he had to wait for me at the top of every hill. We did about 3k feet of climbing total for the day and rode about 38 miles. His fitness level was so much better than mine.
Here we are a couple of months later and I have continued to ride as much as I can, pushing myself as much as possible. I went on my 1st group ride this past weekend with a local club and I did much better than I did only 2 months ago. I was able to keep up with the lead group, we climbed about 4k feet over the day and rode a little over 65 miles averaging 16.6mph which isn't bad considering all the climbing we did.
I feel I still have a lot of room for improvement for sure but my point is, don't give up. Keep riding and add as many miles as you can. It will come if you keep it up (although you'll never be 20 again ;-)