In college I used to regular work with friends who had issues going any decent speed on their bike. We were all broke college students. But these are the things I would check that were the most important:
1.
Seat Height. With your seat to low you are significantly less efficient, and it's significantly harder to increase the speed you can bike at because you're using much less powerful muscles in your legs to pedal with. (Often this could be fixed by raising the seat, but sometimes the entire bike was to small).
2.
Brakes Rubbing on Wheels. If your rim brakes are rubbing on the wheel, you lose a huge amount of power, which (if they're rubbing but not that much) you don't even notice except that you're noticeably slower than everyone else. It's easy to test - pick up each wheel and spin it (for the rear wheel spin it opposite the direction that make the pedals turn). The wheel should spin minimum 3 times before stopping. If it spins less than that take a look at the brake pads and see if they're rubbing against the wheel when the wheel goes around.
3.
Knobby Tires. They're slower than slicks, even fat slicks are better than knobbies.
#1 in particular (seat height) makes it really difficult to increase your speed by riding.