Tire places hawk nitrogen to drivers because it leaks out of the tire much more slowly than atmospheric air. It's also dry, which is a good thing for tires. Nitrogen-filled tires can go 6+ months with no loss of pressure.

Race cars use nitrogen because it doesn't react nearly as much to changes in temperature. I did tire and suspension setup as part of a short-track pit crew for several years. Tires filled with air could swell by +10psi as the tire heated from ambient to +180º. The same tire filled with nitrogen would only change 1-2psi. Higher pressure changes the rollout of the tire, which changes the stagger of the tires (for circletrack racing, the tires on the left are smaller than the tires on the right) which alters the handling of the car.

I'd love to be able to fill my tires with nitrogen. Dry, leaks out more slowly. No real drawbacks.