Male stoker here. We get the same comment regularly and I usually say, "Shh. You're going to ruin it for me." or something equally inane. I agree that it is just an attempt at humor.

The funniest comment we have gotten recently was from someone we were passing on a climb who asked why we had the captain on the rear. I guess the notion that the male should be captain is pretty solidly ingrained in some folks.