personally, I'd go with something that can take 1.5in or 38s , I've ridden a lot on 1.5s and if the paved roads are reasonable and the gravel as well, they work fine. Ive ridden on 50s and obviously on rougher stuff and looser gravel they would help to an extent, but it really depends on the surfaces.
If I were to be reasonably certain the gravel isnt too bad, 38s are great, and on smoother surfaces they are a bit faster than 50s, although again, diff tires can have very diff characteristics riding, so its hard to make a blanket statement regarding width (but I do stand by my experience of 38 regular Marathons vs 50mm Supremes.)

I too have ridden a fair amount on rougher roads on 28 slicks and do appreciate wider tires for comfort due to lower pressures as well as more footprint in looser surfaces. This comes back to the fact that there is gravel and there is gravel, it aint a uniform sort of thing, just so many variables.