Suspension, folding frame, gearing all take a toll on efficiency. Plus, the FS weighs about 29lbs. Your friend's fixie is probably under 20. Tires are also a big factor - you got fat knobbies on your FS, your friend probably has skinny 23mm tires. These are the reasons why you feel a difference in pedalling efficiency.

It has very little to do with size of the wheels, although people argue about it forever.