A data point:

I have a 36" inseam but rode 170 mm cranks for years because they were what came with the bike. Starting with a new bike I had 175 mm cranks mounted. I go back and forth between these two bikes and can't feel the difference. No issues of any sort. I've never tried anything shorter, but I suspect that I wouldn't be able to tell the difference down to, say, 165 mm.

I think I fall into the this issue is over rated camp.

Speedo