I've had different stems on one bike and not noticed any stiffness difference. But then, I also don't notice wheel difference or tire difference. I don't even notice fit difference all that much. Or saddles unless it's a really convex saddle. I do, however notice handlebar differences. The carbon bars I've had definitely flex more. Not bad, just was weird to adjust to back when I switched a few years ago. Like them alot though. And those weird bars with the abrupt bend in the drops are annoying.