Thanks for the responses. In the end, since this is just the initial setup of this bike (trying to get to the point where I can test ride it, at least on rollers), I'm using parts on-hand as much as possible. In the end, I decided to affix the water bottle boss to the hanger with super glue (I will return the unopend red Loctite to the hardware store – that stuff's expensive!). What tension it will see will come from the brakes which will tend to pull it down, so I doubt anything will be going anywhere. Note the "headset spacer" below.

I'll eventually replace this setup with something nicer. Or not. (The PVC will get replaced, at least.) We'll see how it goes.
My apologies for the crappy photography. My basement has horrible light, but it's too dang cold for an outdoor photo session.