I think a bigger question here may be tread pattern rather than just tire width. When I changed tires on my Trek hybrid I went from 38 to 28 and have never looked back with regards to packed gravel, grass, or certainly pavement. Anything the bike can rally sink into, such as soft deep gravel, deep soft sand, or mud will not be fun unless you have deeper tread. Decide how much extra rolling resistance you're willing to tolerate to cover those conditions.