You're stressing the stem. It can crack. Admittedly, instant catastrophic failure in a stem isn't a common occurrence, but I watched it happen to George Hincapie in the Paris-Roubaix race in 2006, and the broken clavicle wasn't nice. And it turned out later that his started to fail early in the race, but it wasn't noticeable.

I had a stem fail on my Rossin last year, cracked on the bar clamp while riding, and while I was able to keep the bars in place and control the bike, the bars swiveled constantly in the stem making use of the brakes almost impossible. Fortunately, I was only six miles from home at the time.

Ended up riding home holding the bars in the high rise position - looked like an expensive homeless persons bike.