Stiffness of the drill press "frame" mechanism is I think a big deal.

the surface where the drill bit engages if not perfectly perpendicular to the bit, the bit will want to wander.

a spring punch can help - lots of effort.
not impossible but... reference short jobber drill bits - less flex- less depth to the flutes making the bit stiffer.