My quess is that it would matter on a mountain bike, but not be critical... It would impact traction and handling, but not enough to make the ride unmanageable, just enough that you aren't riding an optimal setup. If I had a mountain bike tire pointing wrong I would change it.

On a road bike, I have a pair of continentals with arrows, but there isn't enough tread that I would expect it to matter. If there is a difference, I think it would be even less critical than on a mountain bike. If I had the arrow on a road bike pointing the wrong way, I would probably ignore it, until the next time I had the tire partly off for a tube repair, even then I might forget about it.