Ride with fast people, ride in groups. Sounds like you aren't used to riding in a pack. Group riding skills is more important than fitness for you. Don't be timid, my first time out to the local hammerfest, I got dropped on the warm up lap. THE WARM UP LAP! I came back and hung on longer each time. Key to improving was getting stronger and understanding how to ride in a pack.