I ride in to work going pretty slow (about 1 mph than on my trip home) but I enjoy it just as much or more. The interesting thing is that I think I burn close to the same calories even though I'm not getting my heart rate up quite as much. I can't prove that, but that's what my body says to me. This is only a 2 minute difference for me anyhow.
The bottom line for me is how I do on the hills I face. There's one in the last 10 minutes of each of my commutes and it is where I really break a sweat. They only take 4-5 min to climb but they are steep. On days I push it, this is where it happens.