If it really is pea size gravel at most, I wouldn't even bother with changing tires. When tires are that skinny, a little bit of tread won't make a lot of difference. If I was going to do a lot of gravel regularly on skinny tires, I might get something with a bit of a file tread on it, but otherwise I wouldn't bother.
As mentioned already, "gravel" is such a broad term that it really is down to what these particular roads are like. I've ridden a ton of gravel roads in New England that are perfectly good on slick road tires, some in the South that were bumpy but fine, and still others that were all but unrideable. Meaning not necessarily literally impassable, but requiring enough work that it was just no fun. Like I said, if it's the smooth stuff, I wouldn't even bother swapping out tires.