I plan on racing next season but I started shaving this summer. Part of it was because I know sooner or later I'm going to fall, I ride in groups a lot, and ride on some reasonably treacherous roads. I'm not going to lie though, I think it looks better when you're wearing a kit plus I wear jeans off the bike 98.9% of the time, that was another factor.
Anyone who does it to reduce drag is a effing moron.