The problem with my old wheel is that it's a 40 spoke wheel. So I'd need to find a new 635mm rim with 36 holes, and finding anything 635mm is hard in the US.

Will it actually look ridiculous? I guess I'm just not sure how noticeable the 1.3cm difference in diameters (that's what 635 and 622 refer to, no?) is going to be (visually or in riding it).