The folks who pass you could easily be lifetime riders and competitive racers. Or, it could just be that your seeing someone in the middle of their 15 mile commute, after they're all warmed up. 4 miles is barely enough time for me to get warmed enough to sustain any kind of speed worth talking about. Perhaps, even, you are seeing those who like to sight you, gain composure, sprint by like they're awesome, then break down in a huffing fit after they're out of your sight. Hitting 20 on a 4 mile commute is not too shabby; you won't win a competitive sprint but you're obviously no slouch.

Light tires may help; rotational mass requires more effort to move than stationary mass, per physics. Clipless pedals will help tremendously if you're not already running toe-clips. Are you running drop bars?

Otherwise, muscle development and efficiency will come with every mile. When I first started commuting I was on a lugged steel bike with panniers, and struggled to keep up with anyone. Now the bike is lighter, I carry only a few things in a small backpack, and I've learned to pedal and breathe more efficiently. Out of 20 riders I encounter on my commute, I pass 19 of them (although, this probably just means that I encountered 19 leisurely riders and 1 fast rider - or it could mean that those I am passing are either warming up or cooling down; it's all about context).

Overall, unless you intend to flat-out sprint the for the entire 4 miles, then you're not going to shorten your commute enough to worry about. If you were riding 10+ miles then the time savings with every mile of more efficient riding could add up to 5-10 minutes.