You don't have to tear off the foam pads. There is a hole in the middle of each pad (it might be hidden in one of the folds but it's there) that you can stick a 4 mm allen key down into. There is a bolt at the bottom of the hole that fastens the arm rest to the bracket. Loosen the bolt with the allen key, remove and reposition the arm rest and tighten it in place.
BTW, after you remove and reposition the pads, cover the holes with small pieces of electrical tape. That keeps sweat fron dripping down inside and rusting the bolts.