The chain has to bend a bit (probably not using the right word there but hopefully you know what I mean) whenever your rear gear is on anything other than the center ring.

There's a limit on how much the chain can go over before it starts to cause issues.

Right now no one has figured out how to fit more than 11 or 12 speeds in the back. You can't just make a gigantic cassette - if the cassette is to wide it won't work.

I don't really like shifting in the front either. I prefer a triple over a double just so that I don't usually have to shift out of the center ring. I would not recommend going to an igh just for this though, I have an IGH and it's a little slower and clunkier than a regular derailler. It is certainly possible to go with a 1x10, 1x11, or 1x12 setup though. Both Shimano and SRAM sell kits specifically for doing this.