Thanks everybody for the input.

I followed some of the pointers and narrowed it down to actually being a rear hub issue.

After finding more metal around the base of the cassette I ran back to the bike shop. They pulled the rear wheel and found the cassette was climbing onto the splines on the free hub.
further into the investigation we found that the conversion cap had a groove cut into it, and after that the bearing housing on the hub was bad and allowed the whole hub to flex under load.

Most likely what caused the derailleur hanger to crack and was giving me some brake rub under load.

So a rebuild is in order for the rear hub, new cassette, and one last tune up and I hope the last gremlin is gone from this bike.