I think your caliper is loose. You shouldn't have to take it totally apart to fix since it sounds like the mounting nut is what needs to be tightened. The nut behind the brake bridge.

This might help: http://bicycletutor.com/sidepull-caliper-brakes/