Stuff like this cracks me up...and not you, but this thing about clubs, both social and racing where if you don't appear to have "the gear" you are immediately branded.
I love guys that come into the shop and announce that they are "Ride Leaders" like it's an award or something they've earned...LOL...
FWIW, I know a bunch of pretty decent racers that, when the weather is bad put clip-on fenders on their bikes for the very reason you indicated. To keep their nice stuff as nice as they can when riding in lousy conditions. Funny...Zabel endorses a brand of clip-on fenders that will fit on a race bike.
Taking off the fenders and rack...I've told this story before about the young studs that "broke away" on a training ride and were run down by an ex-pro on a cyclocross bike, with cross tires, fenders, rack, and a tent on the rack held on with bungi-cords.
But any weight you can comfortably remove will make your ride more pleasent, unless it's raining and then leave the fenders on.
People need to lighten up...