I've been riding a fixed gear for commutes and general urban riding. I'm surprised to report that I'm no slower than on my geared bikes and it's not particularly hard to ride a fixed gear after a week or two getting used to it. I've climbed all the steepest climbs in my area using my fixed gear and although it's more effort for any given second of climbing the climbs are over a lot faster than if I was gearing down on my other bikes...overall I don't get to the top of a steep climb any more tired than before.

If you've got a single speed and you are not in shape just keep riding it...you'll get stronger quite fast.