If the QR can't hold under power, the wheel will cock and rub against the inside of the left chainstay. This is surprising and annoying, but is unlikely to lead to catastrophe, since it will probably happen when you are going slowly and really mashing on the pedals to accelerate. If you can stand on the pedals and not experience this, you don't need to change the axle position.

Removing the spacer is perfectly OK; just make sure the wheel is centered between the chain stays & seat stays when you lock the QR.