If the headset cups are aluminum, using a drift punch can damage them. The Proper Tool, or homemade facsimile thereof, is much safer.

N.B. if the headset was installed without grease, the cups may have galled in the frame, making removal much more difficult. Be sure to use grease or anti-seize when you reassemble the headset.