I have a nice CF road bike and a steel touring bike that weighs about 10 lbs more. I'm about 2-3 mph faster on the road bike on flat roads. In the hills I'm even slower on the touring bike.
However the touring bike has much fatter tires (700x32 vs 700x25) which along with the wider wheels makes the bike not as aerodynamic. Speaking of aerodynamics, I sit much more upright on the touring bike.
Therefore while I think that the weight slows me down, it's probably only a part of the issue.