It is my amateur understanding that the reason modern bike frames use fat tubes is that the strength of any rod/tube is greatest at the outside edge. i.e. a tube with 1 mm thick walls is potentially stronger than a narrower diameter tube with 2 mm walls... and stronger than a very small diameter (say 10 mm) solid rod.
Therefore, my assumption is that the area you want to remove material from is the worst possible area as far as maintaining strength and rigidity.
With the amount of saddle to bar drop you are seeking, I am guessing you are not an easy riding person, so you could be placing significant stress on the stem. Make your decision accordingly.
Now for the smart alec remark that just came to mind...
If you can't get your back flat enough with that setup, have you considered bending your elbows?