Since you are certain that it is not a cable, technique or shifter problem, or at least not receptive to the idea that it could be such, just get some new parts and try them. You'll need a new cassette someday, get it now. Your derailleur is low end, get a nicer one, Deore or LX. Since you have an engineering background, do the work yourself, it's not rocket surgery. Check the Park Tool site for instructions. Cycling is inexpensive compared to most hobbies, you can buy a lifetime of parts for the cost of one set of autocross race tires! Report back when you get it resolved.