I don’t see an issue with removing the washer if you still have chain clearance from the dropout. You are basically just putting 2 jam nuts together. Without the washer, it might make it harder to separate down the road, but since adjustments are NDS, that shouldn’t matter.

You should put the wheel in a truing stand to center (re-dish) after the washer is removed.

John