My Ti Swift feels the fastest of my bikes. I am convinced it's because it's the lightest bike. Any bit of slope and extra mass slows you down.
On my now-defunct blue Xootr Swift, I sometimes rode with a peleton as training for a big Audax ride, and I found I could keep up just fine except in one spot where there was a long, steady climb, I would tend to get dropped near the end. I recognised that was a fitness issue not a bike issue. On the same ride, in a spot where there is a steeper climb, I would be able to drop other guys again. And in one memorable long downhill, in the middle of the peleton, all of us tucked down, I reached 84km/h.
So I have never felt the bike was an issue, although I can't be as fast on all my bikes. Often people will ask me if the smaller wheels are harder to pedal, mostly I answer no, but sometimes I poke a bit of fun and say ah yeah, much harder.