I bought my 56cm bike about 2 years ago, and have just concluded I bought one size too large, and should have bought a 54. I had a fit done at the time, and they said everything looks good, but I've since tried to shrink the bike. I went from 100mm to a 90mm stem and went from a setback to a 0-setback seatpost. Everything feels pretty good now.

My question is, at this point is there any reason to go down a frame size? Or if everything feels good, just shut up and ride? I don't race, and mostly do gran fondos, centuries, group rides, and commute.