Like a bike frame, smaller diameter means more flexible. Perhaps there's a defect in the rollers so that under load, the rollers are bending causing the bearings to bind. Just a wild idea without knowing how yours are constructed. Or something simple like a spacer is supposed to go between the roller & frame so that under load, the roller is far enough away from the frame so the allowed bending is not a problem.