Some frames are much slower than others. The enlightened ones will argue all day about whether or not this is so and if so or not, why? It is, however, true. It's not really geometry. I've ridden slow and fast road frames with very similar geometry. Though geometry will have an effect, I don't think it overwhelms the other factors such as tubing size, material, joining, etc. Wheels and tires have large effects also.

All that said, very fast bikes can be very comfortable. It's not mutually exclusive at all.