1. If you disconnect the brake cable while the wheel is installed does the drum still drag?
2. Is the backing plate rigidly held, or does it wobble around a bit when the wheel is installed?
3. Is the drag you are experiencing a light rub once per wheel revolution, or enough that the whee doesn’t want to spin freely?
Pictures always help.