Rubber hammers will not do the job in this case. You need either a brass hammer or a "zero blow hammer". The latter is a plastic hammer filled with lead shot which will deliver a very substantial hit without causing damage to the part. I personally prefer the brass hammer.

Doc