I did an organized ride a few years ago with a group from BikeForums (Lancaster Covered Bridges Metric Century), and it was a definite learning experience. I was on a hybrid was over 50 years old and weighed over 300 pounds. I had done rides up to 68 miles the previous year or two, so I figured this metric century would be fun. I should have sensed the impending doom when I went for the easy spin with some people from the group the night before the big ride... I was barely able to keep up.
The day of the ride, I felt good, and kept up with the group on the flats, then we hit the first mild incline and I watched the others disappear into the distance. THis happened at the start, and after each rest stop. When I hit each rest stop, they were waiting for me, so I filled my water bottle and/or grabbed a quick bite to throw into my mouth and take off so that the others didn't have to extend their break, because by the time I got there, they were ready to go.
At the half way stop, I arrived as they finished lunch, and told them all to ride on, because I wasn't feeling well, and would be taking the sag vehicle back. This was completely true, since I was suffering from the early signs of heat exhaustion (It was a very hot August day).
The difference between barely making 30 miles, and completing 68 miles? Riding at a reasonable pace for me... A hybrid rider might keep up with a road bike if the hybrid rider is in better shape than the road rider, but otherwise a hybrid rider could be better served to ride with someone of equal ability.
Last week I did 46 miles with a friend similar to me as far as bike and ability. We started with a group of 30 or so riders. By the time we finished, nobody was around, but we had a great time because we both pushed our limits a bit, but didn't worry about keeping up with the other people who were obviously in better shape than we were.
If you are compelled to push yourself, that is fine, but the bottom line is to do what you enjoy to keep yourself on the bike long term.