This is written for fixed gear, but the same idea applies to any bike without a chain tensioner/derailer. Sheldon says it best:
The chain tension on a fixed gear is quite critical, and is regulated by moving the rear axle back and forth in the fork ends. If the chain is too tight, the drive train will bind, perhaps only at one angle of the pedals (chainwheels are not usually perfectly concentric). It should be tight as it can be without binding. If the chain is too loose, it can fall off.
Notice how freely the drive train turns when the chain is too loose. That is how freely it should turn when you are done, but with as little chain droop as possible.