I commute on a 44-16 single speed. My times on a given ride are much better on my roadie than on the SS and when I take my roadie to work its almost effortless on it. Here's a few reasons why:
-Hills
-My single speed weighs about 29#, my roadie somewhere around 20#
-Better gearing for taking off from a start (I need to toughen up more)
-32c tires on the SS, 23c tires on the roadie (less rolling resistance)

My road bike is just a more efficient machine than my single speed. I ride the single speed to work and grocery getting because it has fenders and its a little more resistant to hopping curbs.

YMMV.