To me your question has two parts, one physical and the other mental.

Riding up a decent grade at a decent speed requires a high power output so it's never "easy" in the physical sense of not requiring a lot of energy (power x time = energy).

Ultimately your speed on hills is governed by your max power output.

How you feel about riding hills has a lot to do with how comfortable you are at elevated power outputs but there's also an element of anticipation.

I'm a born inclinophile, despite my 90+ kg body mass I love climbing hills, I relish the opportunity to strut my stuff (even when the only audience is the wallabies on Flinders Peak)

You appear to be a classic inclinophobe, in fact your symptoms sound a lot like performance anxiety to me*. Dealing with the anxiety may go a long way towards making the process more pleasant.

*I am not a psychologist and you'd be a fool to listen to psychological advice from some random on the intertubes