The biggest factors that affect my commute time aren't related to my fitness or fatigue level (assuming I don't slip too far out of form). My commute is relatively short (4-6 miles each way), and entirely within the island of Manhattan on densely trafficked urban streets, so the routes we take probably don't have much in common. A longer ride will obviously get more benefit from speeding up than a shorter ride will.
But here are the things that make me go faster:
Favorable winds
Carrying less weight
Stretching before I ride
Being late for work