from someone who considers himself to be an average cyclist (ride maybe 2-3000km per year tops) I really dont think this is an issue. Yes, as I mentioned before, I wipe the gritty gunk off my rims when I get in from being in the rain and clean my pads once in a while too, so this helps a bunch and takes only a minute or two when everything is wet (if you let it all dry, its harder to clean)
I realize not everyone likes doing this, but it really helps with braking power and rim life.