Downsizing will also likely have smaller stack so might go against your preferred setup but you can always have the steerer tube longer and add spacers under to raise the stack.

But assuming the stack did not change, shorter reach tends to improve comfort so downsizing isn't bad at all. The only possible problems with downsizing is toe strike and if the reach is too short, your knees might be banging the handelbar.

TdF pros tend to use downsized bikes. I know we shouldn't be copying pros but if you have to downsize for any reason, then go for it.