The fear is that people on e-bikes going on trails on which they are forbidden can affect trail access for all. In some places trail access for bicycles has been hard fought and can be revoked if the powers that be decide to do so.
I'm not saying it's right and I'm not saying e-bikes hurt the trail more than regular bikes or anything close.