It might be the pads, it might be the rims, it might be how they're adjusted, or it might be any combination of the three. What are you using, and what have you done in the way of adjusting? Are the wheels true? How have you adjusted them?

fwiw - most of your stopping power comes from the front brake.