For me, it depends on how much I'm carrying and how far. I've got an MTB with slicks, a rack, and panniers, and I can fly on that thing, but it's nothing like my road bike.

The only part of the bike you need to be concerned about is the wheelset. With a pack and less-than-ideal routes, wheels can take a beating. That's why I built myself a set of 32h/36h 3x Deep V wheels. I've hammered the hell out of them and they're doing great at around 10K miles, including commutes, training, and racing.