You are going to have a much easier time and will spend a lot less money shooting for skinnier, higher pressure 26" tires if your aim is a faster ride with less rolling resistance (although you may end up with a harsher ride).
Yeah the pound factor is definately an issue! Yeah I think your right in that it's probably going to be more hassle and money than its actually worth in performance. Maybe it's just those damn cool couriers on their singlespeeds with giant wheels giving me an inferiority complex!!
Any thoughts on the carbon seatpost/fork issue? Is it worth the upgrade, some of the worse roads can really jangle the spine about!