As said, above, leave it be if there's no problem and the changes are crisp and true. I bet that out of all the teeth on the largest chainring, some sit closer than the 3-4mm you state.