a threaded hole is just about the worst stress riser you can put in a piece of aluminum other than an actual crack. Like I said upthread, if the plan is to put a significant number of miles on the bike, it's a really bad idea.

The whole issue with fatigue failures, and particularly fatigue failures in aluminum, is that the stresses required to start and propagate a crack are much lower than the yield stress. So your mental model of the strength of a component doesn't really do you much good in trying to decide if something will ultimately fail. We know that un-threaded stems have failed, so it is certainly possible.