Just to follow up... I took it to the local shop and they aligned the hanger, and now it's ridable.

When starting the thread I didn't know that the hangers often need alignment or that they could be fixed locally with an alignment tool. Thank you guys for enlightening me on that.

It still does flex when clamping the wheel down, which I imagine could still be a problem if I take the wheel off to change a tire and then don't clamp down with exactly the same amount of pressure. So I did order the aftermarket hanger which will hopefully improve the flexing issue.

Just for fun, here are some pics of the build: