Fact of the matter is the trail, in its existance, has already 'damaged' the environment. The trail wasn't created by nature, it was created by man. And by the nature of the trail itself, it may alter the errosion characteristics of the land on which it was built. Many people kind of overlook this point, but its important.
Now, however, a properly built trail CAN be made to have as little impact to the natural errosion happening naturally. You need to remember that errosion is happening naturally all the time on any type of slope. Some terrain errodes so fast that you can hardly maintain a trail on it anyways. Most concerns that are valid about trail use, and errosion, is that travel on a trail, by probably any means, can be detrimental to the either the trail as a whole, or to its ability to reduce excessive errosion.
People who blame bikes are almost always those not riding them. They think for whatever reasons that they shouldn't see bike tire marks on trails, and get their panties in a bunch when they see a rut in mud, or whatever. Just like the people who complain about horses aren't those riding them. The park people complain about overuse because they actually have to maintain the trails usability, for everyone, and actually somewhat monitor whatever environmental impact they feel is important. Bikers just want a mostly level trail free of obstacles that make them get off the bike.
When it all comes down to it errosion happens anyways. Maybe trails can accellerate it, or reduce it, and over use can positively or negatively affect either characteristic. Any other concern is usually based around greed or ignorance.