Another person who can feel the difference between 22 and 20.
Sure, depends on total bike weight and terrain, but I'd clearly to for the smaller granny.
There will be times as Doug says that you'll be searching for more downshifts.
Take care of these knees.