I suspect that you may have a sticky shift cable. You may have compensated for it a bit by biasing the trim so most gears shift OK, but it isn't predictable. Other things that can cause this are poor hanger alignment, worn RD pivots or pulleys, worn chain, and/or poor RD adjustments .

Try adjusting the RD trim for the most outboard trim that still downshifts (to bigger) OK. Then after each upshift pluck the bare wire away from the downtube like a guitar string, and see if you can work through all the gears this one at a time this way. If you can get it shifting uniformly when plucking after each upshift, but not otherwise, then it's definitely a cable/housing or possibly (but less likely) a sticky shift mechanism