I have heard many times, or should I say I have read (on the internet) many times, that that mold on leather does no harm.

Okay, what do I know. But I just cannot believe it. That mold is living off something. It is getting its nutrients somewhere. Do you think it's livinng on thin air? No, I think it is digesting the leather, or something in the leather. That mold, whatever it is, is shortening the lifespan of your saddle. To my mind, this is obvious and not worth debating.

How bad is the damage? That I don't know. Probably not very severe, and probably impossible to measure, but definitely a bad thing rather than a good thing. My suggestion would be that you clean that stuff away asap and when the leather is dry (like, in the winter) wrap it in a plastic bag. If you keep it dry, mold will not grow.