White water kayaking punches my buttons the most. When the water dries up, I mountain bike because it has a lot of the mental/reaction aspects are similar and it gets you into similar places. When I decided to seriously get into mountain biking, you just have to spend time on the pedals every week to stay in the right kind of shape, and it is really hard for me to do that off road during the week. I rode my MTB on the road for a while, but like Trouble I got tired of having my doors blown off by guys on road bikes so I decided to join them.
I will have to say, once you go over to the dark side, it will detract from spending time on single track. Road bikes are fantastic machines, and if you have any gear head in you, they will suck you in. There is a combination of precision and sophistication of a good road bike that takes riding to another level.