Within reason, there's nothing wrong with it. i have seven bikes, and i think only one of them has the original stem. One of my mountain bikes (now a commuter, with road tires and fenders) has had stems from 60 to 150mm, as I switched from using it for crosscountry riding to downhill to city riding, where I wanted to be upright so i could see the traffic. No problems.
some riders say they have trouble adapting to the steering feel when they change stem length, but it's never taken me more than about 10 seconds. You can get off one bike and get on another that's completely different and not crash. Why should a small stem change be dangerous?
Old-school wisdom says to adjust your saddle first (the fore-and-aft adjustment) to get your knees where they need to be, then buy a stem that puts your hands where you need them.