For Shimano, 105/Ultegra and Deore/Deore XT are the "sweet spot" on the cost/benefit spectrum. I've done most of my riding with these parts, and never felt they were lacking.
+1 I'm aware of, but not overly concerned with weight as I do mostly charity rides, non-competitive organized rides, B/C group rides, light day touring and 30-50 mile pleasure/fitness riding. My touring bike is all Deore and Deore LX including hubs. And I've been more than happy. My road bike (used almost exclusively for group rides and training) has a mixture of Ultegra, Deore, DuraAce, and RX100 (no longer in production) parts. Again, no complaints.
I don't really care about smooth shifting or 3 pound differences.
You should care about smooth shifting. It makes riding a whole lot more pleasant if you can hit the gear you want, when you want it, with a minimum of fuss and clatter.
Three pounds is a lot. You'd have to swap out an entire groupset to save that much. Outside of a crankset or maybe wheels, you will be talking in terms of grams for most component upgrades. You are correct that there are basically four levels of components. Entry level (Sora, Tourney, Acera, Alivio) is functional but usually inexpensively constructed, heavier, and has a utilitarian finish. The Recreational Enthusiast level (105, Tiagra, Ultegra?, Deore, LX, XT?) are better made, durable, somewhat lighter, perform better and generally are a bit better looking. This is a good level for someone who just want to ride lots and enjoy the experience. Then there is the Performance Enthusiast level which is for the more competitive enthusiast who has some expendible cash. Lighter yet, smoother operating, bling level finishes, arguably sacrificing some durability. This level blends into the Pro level where grams and perfect performance mean the difference between finishing in the money and going home empty handed. Big bucks, durability is second to weight and performance (if you win, you can afford to replace stuff often).
I really discourage even new riders from entry level components unless that is all they can afford. The cost to jump from Acera/Alivio to Deore or Sora to Tiagra/105 isn't much in the long run and will add greatly to enjoyment, performance and durability. Upgrading to mid-level can save you money in the long run as mid-level components require less frequent adjustment, are more durable, and general are easier to service or rebuild. For example, when you wear out just one of the three chainrings on an Acera MTB triple crankset, you replace the entire crankset. If one ring is badly worn on a Deore crankset, you can replace just that chainring.
From what the OP describes, I suggest Tiagra/105 for road and Deore for touring or offroad. This should result in good performance at a reasonable cost with a bike that is durable and not excessively heavy for recreational/enthusiast use.