Sorry for the delay. I'll try to answer all of your questions. To determine optimal crank length, I run each client through several balance exercises to evaluate flexibility and mobility at the spine, pelvis, hip, knee and ankle. If a client demonstrates compensation patterns, I use corrective exercises to restore motion to the joint. The optimal crank length correlates to the maximal distance the hip can travel without compensation. Sometimes the culprit is muscle tightness, and sometimes it's myofascial adhesions. In terms of perceived exertion, I have seen clients progress from a maximal effort to a low submaximal effort after one set of corrective exercises. There are reports of 50-100 watt improvements from optimizing crank arm length alone.
While this sounds almost ridiculous, when you consider how powerful inflexibility is, improvements much larger than 50-100 watts is feasible. For the inflexible athlete, the downward dog exercise is extremely difficult (RPE: 10/10), and oftentimes they never reach the end position. In an effort to get deeper, they maximally contract the hip flexors and core (which also usually disrupts breathing), but still fail to reach the final position. With a flexible hamstring, an athlete can reach the downward dog position without much effort. Switching from an excessively long crank to a shorter crank produces a similar change in effort by preventing one or more inflexible muscles from fighting the power phase of the pedal stroke.
Going too short can be a major problem for the competitive cyclist also, which a study identified when measuring VO2 at various crank lengths. The mechanism of limitation is caused mainly by changes in venous return and blood circulation overall. As you start to go too short, you'll start to experience symptoms common in isometric movements like wall-sits and squat pulses.
Finding the optimal crank length can mean a longer or shorter crank length. For those lucky enough to have an optimal length close to the standard 170-175mm offerings, they won't need to investigate this, but I found that there are a large number of cyclists who do not fit within this range. Here are a few rules of thumb that could mean the cranks aren't optimal:
1. The quads burnout quickly during hard seated efforts. (too long or too short)
2. The hips have to be forward in order to produce high power. (too long or the stem is too long)
3. Your natural cadence (without consciously regulating it [turn off your cadence display]) is always less than 90rpm. (too long)
4. In a seated position, the hips always rock no matter what saddle height is used. (too long)