Shifting from larger to smaller on the rear cassette is a friction game. The shifter is releasing the tension on the cable and the only thing pulling on it is the spring in the rear derailleur. You are looking for anything that would keep the cable from allowing the chain to move down through the sprockets.

If you think the alignment is good, which is likely because the chain is moving up the gears OK, here is what I would check:
1. The shifter. If there is anything gummed up in the shifter, it will be sluggish to respond. Clean it out completely and lube it.
2. The cable. Any fray, gunk, or whatever can bind the cable. If you think the housings are clean, replacing the cable is a reasonable thing to try.
3. The rear derailleur. Completely clean and lube.

If the bike is older and hasn't been maintainted, my money is on the shifter. Mine needs attention every 6 months or so.

Good luck,
Dan