As a wheel builder in a cross crazy area I can say with good authority that yes....it can.

Grit and grime/sloppy conditions can help accelerate wear on braking surfaces. Apart from that if the wheels are well built and were applied correctly (eg no one is using their 16 spoke TT wheel for cross) then you can mitigate the damage cross does to wheels.