You forgot how to signal that your pull is over and it's his turn. He was being polite waiting for you!

LEAVING THE FRONT OF THE PACE LINE
AND MOVING TO THE RIGHT

Hold your upper arm at shoulder level with your forearm bent, so that your hand is against the side of your body. This forms an arrow pointing to the right. Hold this signal for two to three seconds to give the rider directly behind you time to prepare to take the lead of the pace line.