My compliments on the organization of your post. You asked a lot of questions! One of my bikes is a CX'er that has been set up in the past as a 1x9... I can answer some of your queries...
You don't need a special frame to build up a 1x9. The secret lies in maintaining a good chainline, which as you know requires that you center the chainring. Do that and you avoid a few problems. Also, your chain will definitely not encounter the chainstay.
I'd personally suggest that you use the middle ring on your current triple (I ran a 42 on my bike with a 12-27 cassette and thought it was perfect). Just remove the other rings, install a chainwatcher and bash guard and you are set. If your bottom bracket is the one that came with the crank your chainline should already fall on the cassette's middle cog.
There won't be a noticeable difference between short and long cage derailleurs. You can use a short cage because you no longer have the 22-tooth difference between the large and small chainrings to worry about anymore, but you don't need to replace your current one if it's working well.
Hubs and wheels will be dictated by your frame. A road rear wheel takes 130mm spacing; mountain is 135. There are frames that split the difference to accommodate both. There are some sturdy road wheels and some decently light mountain wheels... I'd let geometry and comfort dictate my frame choice, not rear dropout spacing.
If you are really going to ride centuries and tour on this bike my strong personal bias would be to stick with drop bars. You can still use bar ends, brifter or downtube shifters. These all work with mtn derailleurs. I would suggest not mixing manufacturers or speeds unless you know what you are doing, though it is possible.
I hope this helps a bit. A 1x9 is a great, simple way to ride. Keep up the diligent research and you will build up a bike that you'll love.
By the way Paul's has just released a beautiful chainkeeper that might simplify things a bit for you. Here's a link:
Good luck with the build. I know more members will be posting answers for you. Check over in the CX forum. A lot of those guys have experience putting together this type of drivetrain.