Need a little schooling on sidepull brakes
 

My 84 Team Fuji has Diacomp sidepull brakes and I've noticed if I adjust the cable so the
pad is fairly close to the wheel after I apply the brakes the caliper on the side of the cable
doesn't retract and drags. If I adjust the pad out a little further I don't have that problem
I just have to squeeze the levers a little further to brake.Is this normal or not?

This was normal for sidepull brakes back in the day. After you have everything tightened down, you'll need to recenter the brake by turning the center bolt toward the pad that's contacting the rim. There's a couple ways to do this:
1. Use a Park Tool OBW-3 to grip the brake spring and turn the assembly.
2. Put a wrench on the nut on the front and behind the caliper. Turn both nuts simultaneously to turn the center bol.
3. Using a flat-faced punch, tap the spring on one side or the other to recenter the brake.

Have fun and welcome to old-school bike mechanicing.

Side pull calipers never center perfectly. The spring binds a little on the caliper arms and causes some friction. Any difference in friction between the sides will cause one or the other arm to retract a little less. You need to adjust the pads a little further from the rim to account for that, and you need to readjust them more frequently than dual pivot calipers. On the plus side, single pivot sidepulls are easier to adjust than other types of calipers.

em

Ive found the now old Campagnolo side pulls have a fairly stiff return spring and Tullo's Hardened toothed washer bites into the frame/fork and back of the brake to stay pretty well centered..

https://rayhosler.files.wordpress.co...ybrakerear.jpg

I have those diacomps on an old giant road bike and I found that adjusting the pads so they're really close and then simply using a 7/16" open end wrench on the forward nut to center the whole thing works really well. This usually has to be done once every season but takes less than a minute. You have to compensate occasionally for the pad wear anyhow.

Before anything else, make sure the brakes are lubricated, including where the springs push against the arms. On the other hand, if the bike is generally worth upgrading, a pair of Tektro dual pivots might be in store.