I had the same thought with my bike. My LBS told me going to a 32 with a little higher pressure wouldn't make much difference, maybe .5MPH. Dropping down to 28 racing tires would be a little more noticeable, but I still do about 10-20% of my biking on a crushed limestone trail and racing 28's wouldn't be the best choice there. I got 2 flats right away with my new bike so had some puncture resistant tubes put in, which means I'll actually be a little heavier and maybe slightly slower, when I was looking for the opposite. As I ride more, I may see that I don't use the trail very much and can go lighter and faster. Guess we'll see.