I think you should ride your road bike, let him set the pace, and when he wants to head back, say you're feeling fine and want to keep going. He either follows you and is worn out or goes back home knowing you are still out there, hammering away.

He won't ask you to ride again and you won't have to be an @ss about it.

I had a similar experience with my son-in-law; although I did ride my mtb. The road bike would've been too cruel and I didn't want to drop him, just let him know he needed to get in much better shape.