I ride through all kinds of stuff on my wheels, a lot of it I would not want on my hands when shopping for food.
Personally I think removing wheels every time you stop is real inconvenience. Is that balanced by the convenience of saving a few seconds with QR skewers during a puncture repair? I lock my bike up at least twice a day, but fix punctures on the road maybe twice a year.