I agree there's some risk involved.

Looking at the photo, it appears (to me, at least) that the tube is not only dented but also slightly bent. That introduces the possibility, however unlikely, of the frame collapsing when the OP hits a few bad bumps, since the SP is a vertically stressed member. As others have said, a seat-post failure is unlikely to be dangerous, but I can never leave well enough alone - I'd try the method I described. But that's just me.

I'd have a pipe wrench handy to twist out the drift, pipe, whatever, rather than leave it in where it adds weight.

You might even get away with using an appropriately sized hardwood dowel instead of a steel rod. Whichever you use, you should file any sharp edges off the end.

My experience with aluminum is that it fatigues and breaks more easily than steel, but it's definitely not brittle. It should survive the operation. The frame should then be stronger than it is now, although it can never be good as new. Body filler and paint would go a long way toward making it LOOK as good as new.