You should be okay for a short time. Short, and depending on the road condition like dusty, grit and wet levels you encounter.
big problem is without the keyed washer-spacer, it will be harder to tighten the BB without causing other probs.