Think of it in terms of leverage. You have your fulcrum at the BB axle, the load at the chainring and your force at the pedal axle. The closer the load is to the fulcrum (ie a small chainring), the easier it is to move for a given amount of force.

So with a big chainring, YOU need to push a lot harder on the pedal to get things rolling. Those with smaller chainrings may be putting less energy into pushing the bicycle along (from a standstill at least), but they are doing more work