They need to suck it up, you are on your own time. Wipe off when you get there. Get there super fast, relax and cool off before they get there. Then wipe down and throw on a fresh tee shirt and shorts over your bike shorts/bibs as they arrive.
I do not mind hanging out in my kit for a bit if I throw on a fresh shirt. Maybe carry a travel size anti stink stick? Before I head out I thrown some gold bond powder or lotion down below, keeps things fresher and less swassy.
What kind of restroom can you not wipe down? Seriously upscale or seriously skeezy? Either way do it out front or on the side of the establishment if not in the washroom.