I say yank the padding.

If you want extra cushion, use Fizik Bar-Gels. They're working very well for me these days.