To save a little wear and tear on my body, I try not to make my commutes to/from work a sprint or workout the way I would on weekend fitness rides.

I have also found that it is a psychological bonus to not always make commutes to/from work "hammer time". That said, I've pushed pretty hard when I've been joined on my rides home from some of my friends that commute on their road bikes but that is a pace I can't realistically maintain.

Not to mention, the bike fit on my commuter converted mountain bike just isn't up to par with the "fits like a glove" fit of my road bike.

With a bike rack and loaded bike bag, it's more about getting from point A to point B safely and with the least amount of stress. I've had a few situations where my slower speed has helped to prevent accidents (e.g. a lady just the other day made a right turn directly in front of me but with my reduced speed I was able to brake and stop with time to spare).

If a slow to medium speed on your commutes helps you be more vigilant in monitoring traffic, road hazards, etc. than the time expense is well worth it IMHO...