You can probably just use washers but it may affect your chain line. I'll let people with more experience chime in.

As for the fork, that's 2mm off on each side. I wouldn't even bother cold setting it, though theoretically you could.