I noticed the biggest difference in my own performance when I went from a hybrid to a drop bar bike. It was an older steel Schwinn LeTour.

Going from that to an aluminum frame only made a marginal difference. Going from aluminum to CF didn't make an iota of difference in performance--only in ride comfort.

YMMV.