The smaller you go the higher the pressure goes the stiffer things get.

Drop down a size, and get a better quality tire. The tires that come with bikes are cheap and usually heavy.

The Panaracer Pasela TG Folding tire in a 35c is going to be smaller, prob quite a bit lighter, and your ride quality shouldn't suffer much.