To be clear, when you say left & right do you mean front and rear?

To shift to a smaller cog/ring, your lever doesn't require nearly as much force, since you are just "releasing" it.
When going to a larger cog/ring, you have to overcome the spring tension of the derailleurs, so it requires noticeably more force.

It also may be a matter of ergonomics. Your hand may be positioned where you don't get the best leverage. That might simply be a matter of rotating the assembly on the bar to get a "straighter push" on the lever compared to your hand position. (BTDT)