you could do this and it would buy you more axle.
it would also mean that the spacing of the hub is narrower (say you removed two 2mm spacers, you'd drop the hub spacing from 130mm to 126mm). this would clamp the frame's dropouts slightly narrower, which is absolutely no problem with steel or titanium, probably only a theoretical problem with aluminum, and likely a no-no with carbon fiber.

as long as you remove the same spacing from each side, the wheel will likely stay centered between the rear brake. But you may not have space on the drive-side (depending on whether the cassette of gears is already close as possible to the frame when the wheel is installed on the bike). And if you remove dropouts from only the non-drive side, you'd need to re-dish the rim. which means a weaker wheel.

it's probably best to just get a longer axle.