That's right- it should be something like:

A mile into a stiff headwind: -10 points
That stiff headwind turning into a tailwind: +10 points for each headwind mile

A long slow climb: -5 points per mile
Coming to the top of that long climb: +5 points for each mile

For me, any miserable part of cycling vanishes after it ends. And every ride is net positive.