I'd say it does turn down the fun level a notch, the more stuff you carry the heavier you are. The wider you are, the more wind resistence you will deal with. I started off commuting with panniers everyday but now I only carry them twice a week.

The Specialized Allez has the eyelets for a rack btw, nice price too.