If you're not going to take pulls, just stay at the back. As the front guy drifts back, let a gap open for him and tell him to pull in front of you. Then you won't disrupt the people in rotation. If you are pulling, as mentioned, take shorter pulls. The bigger the group, the shorter the pull.