Agree with all mentioned so far. Here's my list in order of most significant to lesser so (leaving out stuff like aero bars and time trial helmets, etc):

Tires (high tpi natural casings seem the fastest)
Wheels (lower spoke count is effective in reducing drag and steering responses due to cross winds), with smooth hubs
Tubes (latex seems to roll easier than butyl ones, not sure why)
Chain (keep in top working order and replace often)
Chain-line (not a part, but cross chaining just seems to eat power)
RD - a smooth working RD saves time both in shifting and runs better all around
Rings - some rings (SRAM in particular) just seem to run crappy for me, whereas something like a TA ring runs noticeably better
BB
FD - more in terms of adjustment than the part itself

Saddle of course will occupy a place on the list depending on your tolerance for pain in your nether regions. Same thing for handlebars, if you can stay in the drops longer you'll be faster.