I would say if it is enough to rub the brakes take it in, it will only get worse and potentioly permanently damage your rim. Since that means you would ahve to loosen you're brakes so you would have less stopping power which would be dangerous. I retrue mine if they are a 1-2mm off, but that is me personaly I don't likely wobbily wheels.