Also, how much are you carrying and how? I typically ride with 15+ lbs. in my panniers, and on a heavier hybrid bike with full rack/fenders. I can come pretty close to keeping up with the roadies I see, but there is just a limit to what I can do with that bike.

It is the same as with boxing: a fast road bike, with narrow tires and no cargo with a fit cyclist in the saddle will always beat a slower, wider-tired heavy bike carrying a good about of cargo, even with a very fit cyclist in the saddle.