Back to the title of the thread...Drafting/following close on a MUP is typically a bad idea. There's a lot going on and potential threats are everywhere.

Looking around and being alert should NOT have to include more than an occasional look back.

The "drafting guy" screwed up by not sticking around. OP is right to keep an eye out for him. Can you describe bike/guy? I, too, ride that path most days, though considerably earlier.