Are you getting frequent pinch flats?

The problem with under inflation is pinch flats, the kind with two parallel slits in your inner tube like a snake bit it. If that's not happening to you at 80 psi you're not damaging the rim so I wouldn't worry.

At your weight you might want to consider experimenting with more air pressure at least in your rear tire. It'll probably roll a little more easily at the expense of ride quality. Try pumping your rear tire up to 95 psi. If you don't like it you can always let the air back out.

Over doing air pressure is also bad. It actually makes you work harder to ride the same speed and negatively affects both handling and ride quality.