Besides a bent or misaligned hanger, I'd suspect sticky cables. When the cables don't slide easily you have to over-compensate to get decent shifting in one direction, but it comes at the expense of sluggish shifting in the other.

Try this diagnostic. Adjust the trim as well as possible for good downshifting (to larger sprocket). Now downshift to any middle gear and then upshift one position. If it doesn't shift immediately, pluck the bare wire away from the downtube sharply like a guitar string. If that completes the shift, it means that the cable didn't return all the way before and the housings need to be oiled or possible replaced.

Also, as Andrew S. stated, a worn or sloppy jockey wheel can have the same effect of forcing over compensation to complete shifts. One trick I use to confirm this is to switch the lower and upper pulleys and see if it makes shifting more positive. If so, you can leave it that way for now, but need to buy a new set at your earliest convenience.