addyp, you may want to check out the Rivendell Quickbeam, or grab a used frame like this Schwinn Paramount touring frame I got off eBay.
The Quickbeam has a nice geometry for long rides and is designed to be a "multiple geared" fixed gear bike. Good for varied terrain.
LINK
Older frames have the horizontal dropouts for easy singlespeed/fixed conversion and you can pick and choose the geometry and tube set you like. Paramounts are lugged with Reynolds 531 tubing, very nice to ride at any distance. I think the used frame market will give you the widest number of choices.
addyp, after you've test ridden the San Jose, go check out the Surly Cross Check and the Long Haul Trucker.
You'll really need to ride these bikes to get a feel for what is most comfortable for you.
Or, talk to some custom builders about what you want a bike to do and let them create something for you. I can recommend Bilenky Cycle Works. Take a look at their models, give them a call and talk about what you want. They can easily put horizontal or track dropouts on whatever model you like; stretch a top tube a bit, add extra bosses, etc.