It is done. It has been done. People were touring before freewheels and gears were invented and enjoyed themselves. It would probably be unpleasant in the mountains but not terribly much trouble on the Great Plains or in Holland.

Touring on a track bike, though, would be unpleasant. Steep geometry would get awful tiring.