"why should anyone short of a racer shave down there?"

Well, why would "a racer shave down there?"

Why do body builders, pros and amateurs, shave their bodies? Why do basketball players (currently) wear longer rather than shorter shorts? What's with those silly thongs in which sumo wresters cavort? And those nutty Scots who wear kilts for tossing logs around? Skimpy outfits on triathletes? Tattoos on the arms of football linesmen?

The true reason, OP, as in the above examples, is that shaving your legs exhibits one of the traditional signs of commitment to the sport of cycling, whether or not you race. If you don't shave your legs, grasshopper, you might be a strong rider, you might wear funny shorts and shirts, but your are not quite showing your true and total commitment to the roadie way.