Downshift means going from a harder gear to the next easiest gear. You are trying to shift from one harder gear to the next easier gear which mean the chain has to move from the smaller cog to the next bigger cog. You are getting a delay there. Try turning the barrel adjuster counter-clockwise a little big, a quarter to half turn. That will put slightly more tension on the rear derailleur cable which then should hopefully allow the shifting to speed up a bit.