Whether or not it's worth it this is one example where clipless would help a lot if you wanted to shave some time off your commute. Accelerating hard from every stop will save lots of time and is much easier if you are clipped in and can stand up. That and some drop bars, an aero position, tight clothes and good 25mm tires would allow you to double your distance in the same amount of time.

If you want to get faster in order to ride with others I would extend your commute. I agree with others there isn't much point if you're just commuting 4 miles.