Hi guys, I'm 5' 9.5", I have an inseam of 31.5"

I currently ride a 55cm bike. I'm searching for frames... for a 2nd bike. I feel that the 55 might just be a hair too far of a reach when trying to hold at the hoods. I can still ride in it for 6 hours + without many issues, I just wish it was a hair shorter sometimes. 53 is much too short. I feel if the bike were a 54, it would be a best fit for my reach. Most frames that I like come in 53,55,57 and not so much 54,56, etc.

Perhaps I should get a 55 that I like and just buy a slightly more compact stem to microfit adjust the bike to my liking? I've heard that most of the 5'9 - 5'10 guys ride 55's anyhow.

I'd go for a slightly too small frame and then adjust the stem and saddle height to fit.

Are there any downsides to going with a frame too small? Such as handling or geometry that I should worry about?

Less than going too large. Too big is just that. Not much you can do. Too small? Can be dealt with. I'm riding 6cm too small on one bike. 52cm when 58 cm would be ideal. Took some modding but iti can be done. Take though a less
extreme example. I need 58, I have 56. No problem, taller seatpost, longer stem. I have a 60 and need a 58? Itws just goiing to feel wrong. You can shorten the stem, but can't do much about the seatpost.

Going small you might get shoe overlap on the front wheel just adjust technique.

You trying to keep up with a race Peloton and staying over low at all times, to knock out a Kilometer a minute like those guys ?

For this kind of bike, yes. Racing / Group riding.

Note what the kids racing the ToC are riding ,, lots of seat post and bars with no or little spacers on the steerer..