Look at the base (the part that's left). It should have two flats on it. If so, put a (long) wrench on it and unscrew it. I'm not sure where you can get an original replacement, but if you want to go fancy you can replace it with titanium.
I replaced all four of the pivots on my "winter" bike with titanium parts (along with many of the other fasteners). I got them from Toronto Cycles
Paul Brakes,, Titanium Dura Ace, CNC Bike Brakes.
The only thing you need to do is measure the threads on your pivot so you can select the right one.
No more rust!
Steve