For somebody 6'1" with a 32in inseam that is very unusual... like really, really unusual. But of course you could ride a bigger frame with a shorter stem, since you have a lot of spacers. Most competitive riders with your dimensions would have the opposite problem... not able to get a long enough stem and/or enough drop.

Maybe your "inflexibility" also effects your reach... as in a very rounded back? Is your neck unusually long? My buddy who is 5'9" and has short legs and a flat back, rides a 55 with a 140mm -17 stem. He is barely able to get both the drop and reach he wants with typical frames.

So yes... it's very individual.

Frankly, I think the OP should get a cheap adjustable stem and experiment with different bar positions on his current bike. At least then he'd have a better idea of where he wants to end up.