Yes, the lack of leverage with short cranks is terrible at low cadence. It is hard on your knees and it's hard on the bike and it's just no fun at all. I had 6" cranks on my folding bike a couple years ago and got caught riding home from the station in about 4" of heavy snow. The small wheels (20") didn't help, but my tires were fat enough that most of the time I could ride on the packed slush left by the cars. But I could not go fast enough to keep up a reasonable cadence when I hit the soft stuff. I had to hammer through it, and the short cranks were murder. That was essentially an off-road situation, though. This winter I've been riding 700 x 35c tires with 160mm cranks and it hasn't been a problem.

The important thing, I repeat, is cadence. Short cranks let you spin, and spinning is good. But to spin you have to be able to ride, and if your cranks are too short to ride reliably, then that's no good either.

If you want a rule of thumb, I'd look at it like this: the optimal crank length is the shortest one that will let you ride comfortably through and over the worst obstacles you're likely to encounter. Anything longer than that will impede your spinning at the top end, and therefore show you down when you reach your maximum cadence. On a single speed with a fixed gear, where your cadence may go well over 200rpm on a descent, this will make a difference.