Long distance on fixed gear is good fun and really not that tough once you are used to it. Or I should say that 100 miles is no biggie. I do that pretty regularly, but do not have experience doing longer brevets or ultra distance rides of the type you are thinking about. There is a subset of ultra riders & racers who ride fixed, and if you google the PBP or LEL and fixed gear you'll find a couple of guys who have written about their experiences including bike setups, gearing, tire sizes and the like. Furnace Creek 508 has a fixed gear division and there are some write-ups of that as well.

Bike setup is important, IMO. I can't imagine riding a true track bike over roads for that distance. Or a bike with with the kind of aggressive or urban setup that one sees here and on the street. I have a track bike, but all of my long distance fixed gear riding is on an old steel frame w/classic road geometry and corresponding setup.