Just some random thoughts.
I think it can be done.
Of course, you can race a touring bike too. In fact, the looks you get from either peer group will be parallel.
The key factor is your willingness to live with the restrictions the bike gives. You're quite probably looking at fewer miles each day touring on a racing bike than touring on a touring bike. By this I mean can you ride your racing bike for 6-10 hours each day at 9-15 mph?
Believe it or not, a comfortable, long wheelbased bike, with decently wide tires, will make it more possible for most people to go farther in a day - even unloaded - than a lightning quick road bike.
The balance tips a little one way if you add peers and pacelines to the mix, but crashes hard the other when you're sightseeing and hauling your stuff on so-so roads in the rain.
There appear to be very few absolutes in this sort of thing, however, and one of my favorite tour descriptions is here:
http://www.mile43.com/peterson/NorthRoad/NorthRoad.html
1800 miles, 11.5 days. Seattle to Duluth. Fixed gear.
Great reading too.