At the end of your comment, you allude to the point that I wanted to make: speed is not so much about weight as it is about aerodynamics, and on a lowracer with a tailbox, Peter Heal was about as aero as you can get without a full enclosure. To be sure, low weight helps, just not as much.

As to the question "Why doesn't bike-touring have the kooks like all the other sports do? Ha..." there probably is no definitive answer, but I get the impression that when bike tourists decide that crossing the country isn't enough of a challenge, they go to other parts of the world where the infrastructure isn't as developed, rather than going faster.