Is it really detrimental for the chainline to be off by about 3 to 4 mm on a single-speed bike? I figure it's not that bad since chainlines are mostly not perfect on geared bikes.

Any insight is greatly appreciated.