That would be my opinion as well. When "scuffing" the brake pads, remove the wheel and use a sanding block of some sort to make sure the sanding surface stays square with the pad. Don't remove a lot of material... You just want to break any glaze.
Inspect to make sure the pads are not wearing unevenly; that they are not hitting the rim squarely.
Also, make sure there's no "slop" in the system from kinked cables, worn housing... All that.