You see, some stereotypes are in place for a reason. They automatically judged you because not only do all people judge others on the first impression, but because they've seen out of shape guys on touring bikes before and most, if not all, have left the same impression before.
Nobody in their right mind is going to think to themselves "Wow, if that guy was in shape, he could probably keep up with us!"
This isn't elementary school where the teams are picked at random like in gym class, this is cycling. Those riders go to that ride to get some training miles in and improve their fitness, why should they wait for just some guy they haven't met before?
Did you even get acquainted with the group before setting out? Because if you did (which I'm guessing you didn't) they'd be warmer to you.