I changed from a 700x38 to a 700x28 on my Hybrid bike. I used Sheldon Brown guidline before swapping it. Somewhere down the article near the bottom is where he has a table showing rim size and tire recommendation.
http://sheldonbrown.com/tire-sizing.html

You do have to remove your old tire so that you can measure it. I took a guess at my thickness of the rim instead. Most rims with the original tire that came with it are usually OK to go down in tire size but going up in size is more limited.

As for speed, that is questionable. I did notice a difference from 38 to 28. Not sure how much of a difference for a 32 to 28. Other factor to look at is your weight and the contact area of the narrower tire and how the tire get sqeeze and what max PSI the tire can run. I won't go into that because that's a whole different cans of worms open for debate, but you can find plenty of info dealing with that here and elsewhere.