Sounds like you can handle the pace, but like others have said, it's the accelerations that kills you. That's pretty common, especially if you haven't worked on your VO2 max or anaerobic systems.
Add two types of intervals to your training routine, and you should find that sticking with the accelerations to be easier. As others have said, the speed at which you get dropped isn't the speed that the pack maintains.
Short 30 seconds to 2 minute anaerobic intervals should be of use, in addition to 3-7 minute VO2 intervals. The first are designed to increase your anaerobic work capacity (AWC), and improve your ability to recovery from anaerobic efforts. The second is designed to increase maximum oxygen uptake, and you should find it helpful when the pack accelerates for a couple minutes of hard riding.