It depends on how obsessive you want to get and also the type of rust.

Red rust (Fe2O3) is relatively easy to remove either with chemical or mechanical means, or reverse electrolysis (Electrolytic Rust Removal aka Magic). The electrolysis method works like magic especially on heavy rust.

Black rust (Fe3O4) is harder and also more difficult to remove. Rust also tends to cause pitting of the metal (formation of small pits on the surface) which can be difficult to get into. I use a special stainless steel Dremel brush which is soft enough not to harm the base metal but hard enough to remove the Fe3O4. This works well but you need a lot of patience.

Here is the brush in question:


... along with a low temperature silver brazing rod.

The worst case of rust I had to deal with was a bottom bracket on an old Hobbs of Barbican frame (tip: if a frame is this rusty it's best to avoid it!). I used reverse electrolysis with a sodium hydroxide solution, then the Dremel. Finally I filled the pits in with low temperature silver braze. It took a full month of weekends.

The result:



The yellow patches are the silver. I also had to braze in a new chainstay bridge (stainless steel tube). Altogether a lot of work and it still has to be electroplated.