I never had a problem on 23 mm Grand Prix 4000. But then I don't consider occasional flats a problem. I'm riding 28 mm tires (IRC Formula Pro Tubeless) now on rims that stretch them out to about 32 mm. Much more comfortable, better grip too.

I've always used a smooth tire (like Gatorskin, but not Gatorskin, ever) on dirt and gravel, same size as on pavement. We don't have that much deep, pea-sized gravel here, where you need bigger tires. We mostly have dirt roads, with all kinds of stuff in/on them. Wider is better on gravel but 28 mm is sufficient.