It is harder to control your bike in the aero position, so when you're cruising downhill at well over 30mph it can definitely be hard to maintain control. With practice you will get better at holding your line in the aero position and maintaining stability goes hand in hand with that.