Not rude... pass commuters all the time, if they are slow & holding a line I might not say anything, just pass them with a small wave & a nod. If they have some speed on 'em or I'm not sure what they are doing, I'll say, "on your left" with the small wave and nod. Guy must've just been new or startled.