Well usually aluminum bikes are stiffer because you have to over build
to overcome their inherent lack of fatigue limit as compared to steel.

To get past the fatigue problem you need bigger tubes which give a stiffer frame and you end up with something the same weight as steel but stiffer.
If you use light weight Al tubes you can have a lighter bike than steel, but it will eventually fail because of the lack of a fatigue limit.

I'm not sure carbon handlebars and seat post are going to be all that more comfortable.
I know the bike shops are pushing that stuff but if the bike isn't the right size and isn't set up right you aren't going to fix that by replacing alloy with carbon. That may not be the case with your bike.

If you are happy with your aluminum bike and it fits you, why change?

I agree with peterpan, tires can make a big difference. A higher volume tire with a little less air can smooth the ride out a lot. Many people pump their tires up over the maximum recommended then complain about the harsh ride and talk about switching to carbon.