If you don't care about rolling resistance, that's your choice. But most of us do.

Counter-intuitively, an over-inflated tire is less efficient. If the sidewall can't flex, then the entire mass of the bike is lifted over a bump, dispersing much more energy than would the softer tire. A "harsh ride" indicates that the tire is not absorbing the bumps, and that you're wasting energy.