Maybe you were pulling on a slight uphill? Also drafting position is important in a crosswind, maybe your friend is better at riding in your wind shadow.