Keep in mind, that the brand/model of tire has as much to do with rolling resistance as width. For example, you might not see a difference between a slow 32 and a fast 38.

I've been all over the map with tire sizes on my various bikes. The fixed-gear has 23's, another fixed-gear has 25's, and my cross bike(s) have been between 28's and 35's. I recently retired the 28 Specialized Nimbus tires, which I liked, and installed some Schwalbe Marathon Supreme's in 35c. I've put maybe 500 miles on them now and am still on the fence as to which one works better for me.

The 28's felt faster, but I don't think they really were. The 35's ride completely different. With them, I don't have to worry about road seams or debris at all, I just ride right over it - plus the overall grip is outstanding.

My next set of tires for this bike will likely be the Marathon Supreme in a 30c. This size didn't exist when I made my initial purchase.

-----

As far as the gravel question... I'd say that depends on the size of the gravel, and your overall comfort/skill level on the bike. I wouldn't think twice about heading down a gravel path with the 28's I had (as long as they weren't huge rocks). Anything larger than that should be fine.