Have you tried pulling on the derailleur with your hand, overriding the lever, to see if it still won't move out far enough not to rub?

I ran into a similar problem with a rebuild I just did; new frame, different seat tube diameter so new (Ultegra triple) derailleur. I set the height of the derailleur according to the little guide sticker on the plate, and it just wouldn't move far enough outward. As I was tugging on it with my hand, it finally moved out further, but it kinda "broke loose" in order to do so. WTF? Did I break something? Turns out that at the prescribed height, the inner plate was juuuust touching the tops of the teeth of the middle (40T) chainring. I cheated and raised the der aprx 2mm, beyond the "zone" prescribed by the guide sticker. It's not the smoothest shiftin' triple I've ever had, but it works, and no longer hits. Maybe you're running into (no pun intended) something similar?