Yeah, those wobbles stink. At slower speeds, sometimes I have an itch, or need to adjust my backpack etc. I then hit a stick, bump, etc. Steering goes out of control (one hand can't bring it back in control). I then go flying. This has hapened a couple times over the past year, but havent got hurt at all yet. I guess I am young still, so maybe it's because I know it's happening and am "flexible" enough to land on my feet and sacrifice the bike. Each time I am happy I am fine bit grimace at the thought of my Mariner beign broken. Surprisingly, it has survived each fall without any problems.
At faster speeds I always keep both hands on the handle bar and pay full attention to the road. As people have suggested here, the faster you go the more dangerous it is, but as long as you pay full attention and there aren't any hazards you will probably be ok (there are always risks riding).