I gain 500 feet on my commute in. I average about 9 mph. I'm never in a hurry, especially on 60psi 32c tires. On the way home, about 18 but again, this is just casual riding. I've been thinking about a 28c tire with higher pressures to increase my efficiency a little. I'm an old guy and enjoy just putting along.
I'd say my commuter/hybrid is about 15% less efficient uphill and about 35% less efficient than my road bike at speed or on a downhill.
Perhaps you should stay on the 32mm tires if you're just putting along. I like riding around on my 42mm tires. It doesn't quite flatten the road, but it's not far from it. Then again if you have good roads, then it might not be justified.