My bike came with 37s. I changed the tires to Gatorskin Ultras, 28s. I noticed a big difference on handling; I like it a lot better. I crank the pressure up to about 90 psi. My only caution, though, is to watch out for pot holes, bumps and jumps. I managed to put a dent in my rim, causing all sorts of problems with the spokes and the truing. This all happened after I put on the 28s.
So, yes you will have a rougher ride and yes you'll have to be more careful with different terrains.