Is the damaged area near a seam in the rim? They are hard to see sometimes, but if the dent is on or within 3in or 4in of a welded seam I would not ride it.
Other than that, give it an easy ride at a reasonable pressure and check the spoke tension after the ride. If it's weakened and starting to fail in that area, the spokes should lose tension first.
I would also inspect the area closely for spider cracks in the aluminum coming from the damage. There is a special dye available for showing cracks in aluminum, but if you are that worried you should probably replace the rim.