I'm a 53 year old who likes to ride in the 20-50 mile range, usually with some decent elevation gain and loss, on the types of surfaces you express concern about. Having ridden only road bikes, I inherited a hybrid 3 years ago that was the right size for me, spent a little money to get properly fitted, and I find that even properly fitted, I am too tired and uncomfortable after 15 miles or so in comparison to the road bike, even a road bike that didn't quite fit right and I wanted to replace. I agree that you may find riding more comfortable and enjoyable on a "road" bike.
Be careful as you look, however, in limiting yourself to what the LBS may think of as a pure road bike. Many think of bikes that are suitable for light touring or randonneuring to be in a different classification than "road," even though they have drop handlebars and won't be ridden on anything but pavement, and they may be a huge improvement in comfort and enjoyment over the hybrid yet be better for you and a "pure" road bike. Someone else in this thread suggested a cross bike. And even the manufactureres don't always know how to classify them. To find what is just right for you, you may want to go outside the biggest brands.
For the type of riding I do, which outside of nasty climbing may not be too unlike what you do, I want 3 things. One, tires the right width (which for me is 25-28mm), two, a geometry that is relaxed but not too relaxed, and three, a steel frame for a smooth ride (or in the alternative, a frame that will truly give a "steel-like" ride - from my test ride the Secteur isn't bad for an aluminum bike, but there is still too much shock and vibration for my taste compared to a good steel bike). I looked around a lot a year ago and ended up getting a steel frame Salsa Casseroll in the standard build except with 28mm tires instead of 32mm and a 11-34 cassette instead of the 11-28, to deal with some of the really nasty climbs I have. I love it, I get a glass smooth ride even without ideal glass-smooth surfaces. Sadly, Salsa no longer makes that, and the "replacement" Colossal is more aggressive than I like, designed more as a "true" road bike, and double the price. But one reason I chose the Casseroll may not apply to you in any event. I liked the a triple crank with a 26t small ring, which otherwise was available standard only on "true" touring bikes.
Other bikes I was very interested in included the Jamis steel frame/carbon fork bikes, a Surly (the Surly Pacer is what interested me most), and the REI brand (Novara) bike, especially the Verita. REI's website will itself show you the confusion that bike companies have in classifying bikes. They have different classifications for "road" and "touring," yet the Verita listed under touring says its best use is for "road cycling." The other 2 Novara models listed under "touring" have as their best use "bike commuting and touring." If you find a Novara bike you like, you can often get pretty good deals, when REI has a sale where "everything" is on sale, it doesn't include bikes except for Novara brand, and combined with the REI dividend, the price can be pretty good.
If money had been no object, I actually think my first choice may well have been a Gunnar Sport built with touring gearing, but money was an object for me. I know I ran across at least 2 or 3 other good choices, but without retracing my old research trails or calling the LBS's I went to, I can't remember them.
You may find you are perfectly happy with big brand relaxed geometry bike, like a Specialized Secteur (although you probably want at least 25mm tires), but I would seriously suggest looking around at bikes that are not classified as pure road, and not limit your options to the big brands. One problem with some of these smaller brands is finding models available to test ride, and I was not able to ride everything that attracted me, but if you are really interested, many dealers will find creative ways to let you test out something (one shop got one of the regulars at the shop ride to let me try his bike out, the owner of the LBS I bought from let me ride his Salsa Casseroll).
Good luck.