SPD for anything involving dirt. SPD-SL for my road bike.

The SPD-SL *feel* stiffer and more efficient but I don't have any objective data to back that up, and I don't race or anything so whatever. They feel nice when I win the meaningless town line sprint that nobody else knew we were sprinting for, though!

They're also a little bit more fiddly to clip in and if you miss the pedal you bang the everliving crap out of your shin. If you do end up having to go through any dirt (like, say, to go around the bridge construction after you ignored the "road closed" sign, hypothetically speaking) they pick up mud like nobody's business.

The SPDs are bombproof, less susceptible to mud, and the shoes work better when you have to walk. I've ridden literally thousands of miles in SPDs with no complaints. Sprinting isn't really a thing for me on gravel although sometimes it is on CX but I haven't seen a problem with it. Of course, everyone else is also on SPDs or Eggbeaters so it's kind of a level playing field in that respect.

I wouldn't have a problem with using them on my road bike but since I already own the SPD-SLs I might as well use them.