I think even I could flex the wheels/frame enough to get brake rub with only 1mm clearance. That seems way, way too close. Also, I'd be curious why you say the wheels aren't flexing. go out to your bike and push the wheel over with your thumb. Can't you get it to move? Don't you think rocking the bike back and forth while climbing is causing some wheel flex?