Rims should last a long time, they do wear from braking but any rim that you get less than 10k out of probably fell victim to a brake pad with lots of embedded crap in it. It's a pretty pernicious process, you get sand in the pad, which then galls up aluminum, and that aluminum galls up more aluminum.
I'm guessing your surface is now kind of dished out, which is why your brakes don't really work. You're putting a square peg in a round(ed) hole, as it were.
Good on you for replacing them once they get worn, as the failure you get when they wear too thin is catastophic. Some people get lucky and it just sort of separates, but my buddy has a wheel from a guy who had the thing peel off like a ribbon, jam in the brake, and send him on a big endo.