This could just be your perception. Faster riders are going to pass you, but you'll never encounter others riding about the same speed as you.

To increase speed takes a lot of small changes to the bike, including reducing windage (bags, body position), smooth tires at the right pressure, clipless pedals, reducing your and the bike's weight, etc. But the one big change is your effort and fitness. You may consider yourself fit, but it takes years to develop the strength and aerobic capacity to sustain long hard efforts. I ride essentially a hybrid, and keep up with or pass most road bikes on my commute, but this didn't happen overnight.