I've been riding about 2.5 years, and am still rapidly improving. There is a big difference between 'just riding' and 'training with a purpose'. For the first year, I just commuted to work 8 miles each direction, on a hybrid. I lost maybe 5 pounds, then gained it back when it stopped being a huge effort to ride the commute. I wanted to go faster, so I got a road bike, and then started doing group rides. At first, my goal was to finish the group ride while there was still someone in the parking lot. I finished long after they did, but I always finished. I finally started finishing with the B group, then started riding with the A's, and eventually started finishing with them. Then I found a faster group, and only ride with the old group when I want slower miles. I got a coach after the first year on the road bike, and started actually training, and eating to fuel that training, instead of training so I could eat. Over the course of about 6 months of that, I dropped 30 pounds, which of course also made me faster. I started racing, and found I loved it. Most people here would consider me pretty strong and fast, but I am still constantly setting personal bests as I pursue those (of all ages) faster than me. Age affects us, but not to the extent most think it does, so long as we have the right approach and don't let it automatically lower our expectations.

So, if you want to go faster, then train yourself to go faster. That includes lots of base miles, and also hard intervals. You do have to teach yourself to ride fast, by riding fast. Once you can hit over 30 mph on the flat, solo, you will find it easier to ride in the mid 20's, and you'll be able to more readily hold your own in a pace line. Hard training isn't for everyone. I'm in the off-season right now, and only riding 6-8 hours/week. In another week, I enter my base training phase, and that will jump to 14-16 hours/week, with gym work on top of that.

Improving is a combination of both how much time you spend training, and how long you've been training. Some say you don't max your potential until you've been at it (hard) for 7 years. Some say 4. It is most certainly longer than 2 or 3. The strong riders have been at it awhile, building that base of strength and endurance.

Ride lots, ride hard, and the speed (and endurance) will come.