If you go, the worst thing that can happen is that you find out there is someone at work worth spending some exercise time with outside of work. Then maybe you won't seem like the unabomber at the office.

Why don't you find out what he is really good at, and then do that with him after you've ventured on this epic 30 mile trek you are considering.

I am a below average cyclist and went cycling with a group of guys at work. They dropped me and laughed at me later. I got a couple of them to go swimming with me a few days afterward, and practically had to save one of them from drowning in 3 1/2 feet of water. He doesn't bring up the story of "dropping" me on the bike anymore.