The advantage to having your feet attached to the pedal (clip less bindings or toe clips) is that it allows you to "pedal in circles," meaning that you can apply force to the cranks throughout the pedal stroke and not just on the down stroke as you would do with regular pedals.

If you're going to be clipped in, most people like the clipless pedals that act similar to ski bindings because you can quickly step in and twist out instead of having to reach down and tighten a strap. The term "clipless" here can be confusing because you're actually clipped in, but it really means you are not using an old style cage and strap around the foot.

The disadvantage to being clipped in is the cost of the pedals and need for cycling shoes with special cleats for the clips. That also means walking around in cycling shoes off the bike, which may or may not be an issue. If it is an issue, there are more "walk friendly" shoes available. The downside to many of those is that they flex, which is good for walking, but less efficient for cycling. Another disadvantage is that there is a bit of a learning curve to getting out of the clips and it's not uncommon for new users to fall over at stoplights.

The pedals that Machka refers to are an excellent way to have both styles in one pedal. I use something similar on a bike for around town.