Think about it like this. A track bike is the most minimalistic bike you can ride, right? Well, when you're that stripped down to the essentials, everything is going to make a little difference that could buy you a little time in a race.
Also, actually racing your bike generally merits higher quality componentry anyway. The trick is figuring out which high quality components benefit someone who just appreciates quality vs. someone who is racing.