This is driving me crazy.

I have a dj style bike with only the bottom and middle front ring in use. The top ring is a chainring guard. I had to replace the bottom bracket and the crankset. The old BB was an Octalink 118 mm with matching crankset. The new setup is a square taper crank 113mm with matching crankset. The chainline seems fine, the middle of the rear 8 speed cassette lines up with the middle chainring, plus I did a bunch of calculations to make sure it was correct.

Anyways, I have tried 2 different dérailleurs and I cannot get the chain to not rub on the last 3cogs of the cassette when in the small ring in the front, or vice versa. I would expect some rub on the small/small but it does this on the 6/7 gear as well. If I adjust it any more, it will rub on the top cogs. The current dérailleur is a 9 speed front. There is just no way I can set it up any nicer, it is perfectly in line and as low as it can go without smashing the chainring guard.

Any idea what could be up here?

Inner 2 of 3 you actually need a longer BB rather than a shorter.
the middle space between the 2 chainrings is chainline center on a double.

you may need someone who can see 1st hand what you are talking about.

I ran a 2 of 3 double in the 80's , no issues .. 52-36
chainring guard just a bit bigger than the outside chainring..

so FD height was about the same as if I had the 52t on the outside
rather than the middle.

Well the front chainrings are tiny MTB rings. All I should have to do in theory is stop it from shifting to the top ring.

This could be the problem, although I have run a triple as a double (inner-middle-bash) without changing the bottom bracket, and it was fine. Can you post photos?

How many speeds is the cassette and what speed is the chain?

Sounds like your BB is a bit short.

If you don't want to buy a longer one, you could try tweaking your derailleur cage (not easy to do right).