This is a debate I have all the time when I'm buying a bike. I'm exactly 5'10", which for essentially every manufacturer and every type of bike means that I'm right between a medium and a large. Given that, I've bought both large sized bikes and medium sized bikes based on what I'm using the bike for. For a performance road bike, I went with a medium. By lowering the stem (a lot) and exposing more seatpost I got a better race-type position than would have been possible on a large while having more comfort (due to more seatpost) and sharper handling (due to the overall smaller frame). For my gravel/adventure bike, I went with a large to get a more upright/roomy position, which works great for more stability and relaxed positioning. For a trail MTB, I went with a medium because the trails around here are super twisty and tight and getting a larger bike around would be a pain. If I were riding wide-open trails fast, I would have gone with a large trail bike. Etc. So yes, going smaller sometimes makes sense to me, other times I go with the bigger option.