More air in your tires will not necessarily reduce your rolling resistance, especially on rougher surfaces.
Feel and hear that buzz? That's energy that has to come from somewhere. The only source of energy on a bike is your legs, so that buzz will manifest itself as increased rolling resistance.
Besides, rolling resistance is pretty minimal to start with on even halfway decent tires.