Tires. Good tires are lighter where it counts (on the outside of the rim, so fewer grams have more of an impact on rideability of the bike) and handle and ride a ton better than old and/or cheap tires.

Lighter wheels make a big difference too.

What kind of racing? You may be able to get away with downtube shifters in time trials and tris (they are lighter anyways,) but integrated shift/brake levers will be your friend in criteriums or road racing especially if you are just now learning how to use downtube shifters. Start looking into aero bars or aero clip-ons if you are doing tris and/or time trials.