i've had that happen to me twice, luckily both times were when i was just starting pedaling and I was able to jump off the bike real quick and carry it off the street

one reason why vertical drop outs are nice :-p

on both of my bikes which i've built the crank arms ended up getting loose.. kinda made me worry about some bikes i've flipped