>>>>After a couple hundred yards I realized something, I was right, he smoked me.<<<<
I had just the opposite happen today as I rode my mountain bike home from work. After crossing a major turnpike, I suddenly met up with a guy on a road bike, all decked out in colorful spandex gear. I figured that if this was the first time I'd become aware of him, he must have sped like heck out of nowhere in the few minutes it took me to get across the road. Unwilling to try to pace with this guy after a hard day at work, I waved him to pass me, but he shook his head as if to say "Nah, don't worry about it." Next thing I know, I turn back and the guy is about half a mile behind me, pedaling his butt off and getting nowhere fast.
I think it's about legs, but also about technique and, mostly, about the mood you're in when you're on the bike. If there hadn't been a great song coming over the speaker on my handlebar at the time, I might have easily been the one left in the dirt!