Two wheel sets, four tires, two cassettes... It gets expensive and a bit of hassle. I swap wheels on my gravel bike - but for the tires, e.g. 28s to 40s. Unless you're routinely doing solo TTs, I'd skip the 50+ wheels and stick with a light "all around" profile in the 38-42mm range. One wheel set.
I have a couple sets of 41-42mm wheels (on different bikes) and love them. These Reynolds AR41 wheels show up on sale fairly often - still available for rim brakes.
