You dont need a new wheelset, much less two new wheelsets.
And 23mm tires?...its 2023 man, plump up a bit.
Anyways, yeah if you were almost blown off the side of the road with wheels that arent even 25mm deep, then you will really feel the crosswind with 50mm wheels. But get em if you think the performance gains are worth the cost.
As for buying a second wheelset...why not just keep the wheels you have and use them for the times when you think 50mm wheels arent a good idea?
All this is really moot since you are already the fastest person in the world since you are going like 45x the speed of sound. ---> I could only inch forward at about 15km/s