I don't think that's really the question - you aren't asking us how to gently brush off your co-worker are you? I'm guessing that the crux of it is, how can the avid cyclist be a slowpoke compared to a new cyclist and a random cyclist?

He might just be slow and won't admit it to himself, but I'd be cautious about that. He may actually be out of shape as he said. So don't worry about it - if he is really slow he probably won't ask to ride again, except for maybe a casual, social ride as someone mentioned. If he's just out of shape, he'll probably catch up with you and it could turn out being good rides.