Weld/Build quality asside, theres no reason to think that a lower end component group will wear out quicker than say Ultegra or Dura-Ace. In fact, especially between Ultegra/XT and Dura-Ace/XTR, I would say that the Ultegra/XT would last longer than Dura Ace/XTR which is more geared to just shaving grams and being lighter at the expense of more money and less durability. Ti cogs versus AL cogs, etc. etc.

What you will notice is how fast your BA shall feel after riding the heavier bike, I commute to work on a MTB with a pannier. The few times I ride my road bike with a much smaller and aerodynamic pannier (more like a trunk bag, rather than a pannier), I feel like David Millar in the '03 TdF prologue (sans the chain problem ) so after a couple hundred miles in the Windsor, taking the BA out for a spin will feel fast (and probably will be faster)

Jay