OP did not do anything wrong, at least from what I read. maybe the "cervelo dude" have seen something on the road, like a rock or a pot whole?
Anyway, I would ask him to clarify.

I was riding with a group of complete strangers, sometimes they draft after me, sometimes I grab a wheel of somebody else, I have been mercilessly dropped too, so what.

PS And wow, your average speed in slow ride is my average in fast day, I never had (so far) average of 20 mph.