My concerns are:

• passing at unsafe speeds
• trail erosion

Speeding is not an e-bike issue as I've seen many a roadie going top speed down a busy MUP. Enforcement is tricky because, while it happens, it's not a big enough issue (at least where I live) to dedicate police to the task of bike trail speed enforcement. To my mind, the power/speed differentiation between an e-bike and a moped is sufficient make ebikes compatible with most bike infrastructure. There should still be enforcement, but the enforcement should be because you operated your vehicle unsafely, not because of what technology you used to accomplish it.

Trail erosion seems like it'd be largely an issue horsepower and weight. Really weight probably isn't that helpful of a factor given that the weight of an ebike is generally less than the weight of the rider. So it comes down to horsepower and responsible trail riding in general, which is not ebike-specific. Identifying what constitutes damaging trail behavior would probably be more helpful than just targeting a specific kind of vehicle.

If you are operating your vehicle safely and not to the detriment of the infrastructure or other users, I could care less whether you power it by your legs, your arm, electricity, hamsters, or Mr. Fusion. I don't see why it's any of my business.