Does it make a difference in average speed ? Not enough that I have noticed it.
Interfere with panniers...etc. Not much.
I have used them on a couple long tours. Overall for me they aren't as big of a deal that I feel strongly either way.
They are kind of a pain to deal with when packing a bike for shipping, not a big deal but a bit more trouble. They tend to rattle a bit. There is always the chance that the odd stick will jam in them possibly breaking the fender or even crashing the bike. They are a few more bolts that can potentially rattle loose. The biggest problems I have had with them was in conditions where they clog up and rub the tire. That has happened to me twice on tour once with mud and once with tar and gravel in 100+ F heat where the over applied the tar on some fresh chip seal. It actually stopped our wheels from turning and required digging it out with a stick more than once. Not sure it it would have been worse having the tar on the rest of the bike and gear.
Personally I never found it to be that huge of a problem to ride in the rain without fenders so while I have them on my heavy touring bike (which I seldom use any more), I have had no desire to put them on my lighter bikes (that I tour on since I started packing lighter and lighter). I guess I'd say that for me the cons slightly outweigh the pluses for touring.