You went on a charity ride and you were shocked to find that not everyone was following the rules for closely packed peloton racing? You were passing someone on the right, without warning from what you've said, and you're surprised he came right into you?
You mentioned teams, are these racing teams or fundraising teams? You mentioned starting with your team in the back and then working you're way to the front to ride with buds who apparently aren't on your team, were you planning a breakaway? Setting tempo to wear out the other teams? Would your original team members be bringing you waterbottles from the directeur sportive's team car?
I'll just reiterate the rules of the road as they pertain to cars, bikes, airplanes, and ships at sea: the overtaking vessel is responsible for safely passing the overtaken vessel. In this you failed both by not signaling your intentions and by passing on the wrong side. But your fundamental problem was mistaking a charity ride for a road race.