Stop being a wuss.
Hanging with a pack is all about confidence in your ability and your ability to suffer. If you want to ride with the big boys then you will have to hurt. Think it's easy for them? No, the difference is in attitude. If you are training half as hard as you say and as fast as you say then it should be no problem.