If the 32 mm tires you replaced had a heavy tread or even small knobs and you replaced them with lighter, smooth 28 mm tires, you certainly would notice a difference. Replacing smooth 28 mm tires with 25 or 23 mm tires will probably not make a great difference. Lighter tires and wheels are indeed an advantage but they aren't magic.