A 31mm increase in height is more than what my bike would allow for purely using spacers in total (25mm in spacer stack height). I dropped my stem 5mm to get more aggressive and it made a pretty noticeable difference across the board (in terms of arm stress, back stress but thankfully torque as well). It's really difficult to make this call from the internet, but I would personally go with the less drastic change. Assuming that you're in good shape, and this bike is the right size for you, it shouldn't take a crazy adjustment to go from "pretty uncomfortable" to "I can manage this for now". You're bound to get stronger if you keep ramping up your rides.

I'd take a hard look at saddle position before making an adjustment as drastic as a 31mm increase in height, if for no other reason than the fact that you're deviating significantly from the position the manufacturer intended. Since you're a pretty strong rider already, I'm inclined to believe the stem may not be the issue. But I'm also a noob to biking, so I'd still err towards what the fitter says.