Yeah, I would just relax and enjoy the ride. That being said, I have a 4 mile commute as well. I am in the Ozarks so I have good hills on the commute. I usually don't worry much on the way in and just enjoy it, it takes me between 18-20 minutes usually. On the way home I play more. I got home yesterday in 14 minutes flat. The thermometer on the bike read 101 F and I'm riding a Surly LHT with bags and junk in them. I never seem to empty them so it isn't a lightweight bike. I'm running 38s as well. I use platform pedals with studs, so you can pull up a bit, but I don't usually.

Enjoy your own ride and don't worry about anyone else. If you feel like pushing it, do it, when you don't, don't. My ride is one of the best parts of my day, no matter how hot it is. I love it.