Gunk gets stiffer in the cold. Moving to smaller in the rear just uses the spring tension, moving up you are pushing the cable, so this is indeed mostly noted when moving down the cassette.