I couldn't read all the replies, but what gear ratio are you running? If it's something like 42x16 that isn't going to cut it against a multi-speed bike that can go up to 52x14 or whatever if they want to, whether or not they have fatter tires.

As far as serious riders being fast, it's true. Whether or not they look goofy in spandex, and whether or not they ride a cool fixed gear, if someone trains daily (and I'm not talking about a ride to the store with a cigarette break halfway) they are going to be pretty fast. A friend of mine who races and trains a couple hours a day on rollers can of course kill me and he is only in the amateur racing class..

-Jason