I frequently ride a short section of crushed gravel trail on 25c tires with no issues, but I definitely slow down and ride as gingerly as possible. The gravel is angular and not very fun to ride on with 25c tires. I can't imagine doing it for more than a few miles. Same bike with 33c knobbies floats over this stuff like it isn't even there.
As someone else said, a lot of this depends on the type of road. Hard-packed dirt or gravel is no problem for 25's. Doing 40 miles of loose gravel, rocks, holes, etc on 25's sounds brutal.