I suspect you'll notice a significant difference in speed by switching to 23c. All things being equal, they will be lighter, offer less rolling resistance, and be more aerodynamic. The tradeoff is that you need to run them at a higher pressure, and they will be harsher on rough roads. Personally I don't care for 23s on most "real" roads, but they can be ok. I generally run 25s or 28s depending on the situation.
I don't think you'll have a problem using your 28/32 tubes in a 23c tire. Just be careful when mounting it so that you don't pinch the tube somewhere. Right now I have a 27x1 1/4 tube (32-ish mm) in a 25c tire with no issues.
Regardless of what tire you buy, get one with a smooth tread. Tread patterns are useless on pavement, and will only create noise and make you slower.