Remove the yoke from the crossover cable, thereby isolating the brake completely from the lever cable. Now try applying the brake by pulling just the crossover cable. If the return is sluggish, then there is a problem is in the brakes (there may still be another one in the brake cable). If the return is crisp, then the problem is between the levers and the brake.

If it's the latter, one often overlooked complication for rear centerpulls is the cable arc leading to the rear hanger. If it's too long or too short, the inner cable will rub against the outer cable. A more obvious complication with rear centerpulls, is that the inner cable must not rub against either the hanger or the outer cable as it leaves the hanger.