Captain, I agree with you in part, but I've been doing essentially the same commute for 5 years. I added a cyclocomputer last year and going fast, trying to beat times, or maintain a certain speed added a new aspect of fun to my commute. Do I do this every commute? Heck no. Some days just poddling (?word) along is great too.
I haven't found the effort I expend on my commute affecting my job at all.
In response to your "what's the rush to get to work," I usually take it easier going into work so I don't show up a sweaty mess. The ride home is, more often than not, a time trial.