I'm about that height, it is essentially a super easy gain in terms of hip flexion or whatever you call it. Either more aero at same hip angle, or more power at a better angle.
I had 175's and went to 165's. It's a legitimate and accepted change to make.
IMO if budget on selling/buying your cranks to swap is good, go for it. Also, if you have more money to spend, a really low stack pedal setup like Speedplay Zeros can get another mm or few over SPD-SL's.