Its normal. At least on mine it is. The chain is dropping down to a smaller cog and the ramps make it go quick. As such if I have pressure on the down shift it is like slamming the chain in the cogs. And with the smaller cogs the chain is less likely to be engaged than if it was on a larger cog.
Or imagine this. You are responsible for moving a 4X8 sheet of plywood up and down an 8ft tall wall all by yourself. When you move the plywood up the wall you are under it pushing it up, it is a slow gradual climb up the wall for the plywood and when it reaches the top level it slides into place easily. But once on top when you have to move it down there is nothing to catch it at the bottom of the wall since you have to be on the top with it in order to push it over the edge. It drops to the next level rather quickly and loudly instead of gradually.
Listen to others and if yours is worse than others then maybe look at it closer.