You know what? Screw it. Wear the polka-dot jersey. Wear what you like. You aren't racing the TdF, as a matter of fact the TdF isn't even running right now. If the uptight "racer" clique can't handle it, well, that's their issue.
And besides, if anyone does hassle you? Just tell them you ARE the best climber in the Tour de TJHOO.