I don't weigh much so I haven't had problems with the stem yet. I do plan to replace that soon though. So far I've changed the chain, crankset, BB, pedals, clips, straps and got a new freewheel. Wheelset, stem, bars, and fork are next. I find the stock seat pretty comfy so that will stay for now.
I found that upgrading the drivetrain made a noticable and enjoyable difference.