"The shaft is located in a sleeve of the fixture which should isolate most all of the side load from the drill press arbor bearings. No?"
It would be relatively easy to isolate those loads if you wanted to with a drill press. One could mount a bearing or two on something and take the load off the quill or shank to the column or the wall,etc... It would also be possible to mount the hole saw on a spindle that took the load off to the table, though that would limit the depth of cut. Not necessary really, but possible.
By the way, just to dispel a little of the panic on side loads here are two videos on milling in a drill press. Waste of time in a a KIjiji/craigs world with virtually free milling machines in every dumpster...
http://smartflix.com/store/video/90/...he-Drill-Press
http://smartflix.com/store/video/43/...he-Drill-Press