Changing your stem for one which is 40 or 50mm longer will cause a significant change in handling. What we need to establish is whether it would be a good change or a bad one.
Regardless of your dimensions or position or geometry of the bike, your steering will slow noticeably. Now, if the 80mm is already too short and twitchy, going longer may be just what you need. Are you cramped in all positions with the 80mm stem or are you just not used to bending your elbows? However, consider that a) you will be potentially stretching further over the front, increasing the angle between your torso and arms which will require a little more core energy to support over time and b) although rideable you may not be able to bend your arms enough when just cruising which could tire your neck and shoulders.
I find there’s quite a fine line when selecting the ‘right’ stem length although most people can adapt within a range of about 30mm.
Without seeing you riding the bike and assuming your current fit is in the right ballpark for the sort of riding you do, I would suggest erring on the side of caution and moving in 10mm steps.
Over a long ride I think you’d be surprised what difference it can make.
Obviously it would make sense to buy cheap or used parts until you dial in exactly what you want.