Because you will not get the intended strength and stiffness - in other words - the wheel might fail.

No one will knowingly accept the liability of changing the design for you. We are all going to say to follow the manufacturer's design, or come up with something where you and only you are accepting the liability.

More technical: the wheel bearings are angular contact, as the angle is changed, the radial and axial stiffness and strength vary. Cups and cones are manufactured to work with a very limited range of ball sizes and ball track radii. Depending on how the particular cup (or cone) is made, catastrophic failure may result in using the part outside it's intended design.

Smaller balls will result in higher stress concentrations. Sufficiently larger ball (<~10% oversize or more) will not mate properly with the cup/cone surface and may have multiple wear tracks. One of these might be a rolling track (good) and the others will be sliding tracks (very bad).