If the spoke threads are bottoming out in the nipples, yeah that 4mm across is that critical. Also check your lacing pattern very closely to be sure - you may have done a 2-cross instead of a 3-cross, or something else weird.

One idea if you're using long nipples is to try shorter ones. But too much spoke through the bottom of the nipple could poke holes in your tube.