Not really. Would you say that some of the best tennis and football (soccer for you silly people) players out there are "not fit"? They do get a cramp during matches every now and again.
Strangely, I don't remember seeing cyclists get off the bike with a cramp.

Anyway, cramps come with exhaustion and losing minerals and whatnot. I guess a sudden bad cramp at high rpm's on a fixed could make you crash... but with muscles that tired you probably couldn't get up to high rpm's in the first place.