Hello all from England!

Brought a hybrid recently with 700x35 tyres, I'd like to get a bit more speed though and was wondering if I brought 700x32 or even 28/25 (as 700x35 slicks are rare and just too big) how that would effect the rim, would it be suitable to buy smaller tire width? Many thanks everyone