My Specialized Sirrus is a lot slower than the Allez I used to own. About 3mph slower over 10-15 miles. But, guess what? I am way more comfortable and enjoy the Sirrus much more. Could I make it faster? probably. Do I want to? nope. All I plan on changing is the rear derailleur to a Deore or LX, and the chain to an SRAM. Maybe some lock on ergo grips as well, these tend to move while riding, but are so comfy.
IF you really wanna go faster, you gotta buy a race-oriented road bike.
I will stick with my comfortable hybrid. I may go for shorter rides, but I ride more often, which is more miles in the end!