Right. By removing the stress on your knees, you're also removing the speed when you climb. That, right there, is the difference.

When I ride single speed I find that I usually climb much faster simply because I need to in order to survive the climb, in a gear that's really too tall for the hill. A lot of standing up, a lot of mashing, a lot of grunting.

Also, when I do group rides on my single speed, I almost always pass people when climbing (most of them are on geared road bikes). Same explanation.