I usually don't care, if I don't sense any danger.
If I'm riding my local MUP when squirrels are particulary active (read: out in force and nuttier than normal), or the person who sneaks up and drafts looks like a complete goofball, then I would probably "encourage" the person to get off my wheel.