Bikes are meant to be ridden. You may get more paint chips from gravel and your consumable parts may wear a bit faster from the road dust, but that's about it. Maybe some cable slap on your paint, depending. If things get muddy you could see issues - like the RD getting stuck with mud and ripping free. General gravel riding is fine though.