If the wheel is built well, you're probably fine, but if one spoke breaks then it'll go way more out of true than it would with more spokes. It'll just make it a lot more of a PITA to get around if you do have breakage.