Perhaps that person's idea of the right frame size is different from yours. I don't think this has anything to do with the size of the wheels.
When I built up my 650A "rando" bike, I used the same frame size as I had with 700x28s, knowing that I'd lose a little height. Came out perfect -- I spend very little time with my crotch on the top tube anyway.
