I don't know if you should worry or not. It depends on what margin of safety you're comfortable with. Obviously you can load 70 pounds on the bike and ride it out of your driveway and the wheels aren't going to immediately collapse. The spokes are in tension and it takes a lot more static weight than that to pull the rim apart.

You can't predict road surface conditions for roads miles away from your home that you've never ridden before. I just rode 600+ miles around Lake Ontario and I encountered road construction, a bit of gravel temporary road, and lots of cracked, bumpy roads in rural areas. In populated areas, I rode in traffic and was often forced to bounce through the roughness of drainage grates along the side of the road. (It's not the grates themselves that cause the problem, but the inevitable crumbling road and dip that surrounds them.) There were lots of time I was very happy to have my 48-spoke wheels with double wall rims. A lighter wheel might or might not have been damaged, but I didn't have to spend my whole tour worrying about it.

Your wheels and tires have to absorb the brunt of every sin the road surface has to offer. I prefer a wide safety margin there so I can worry about more important things when I'm touring, like where's the next ice cream stop.