I don't think the Gossamer is much of an upgrade from any crankset. It's a bit of a pig. But yeah, you're not going to notice a performance gain from a crankset IMO. Only benefit you get from a new crankset is bling and weight loss, though the Gossamer will likely provide you neither.

If you're going to replace stuff, replace it with top drawer, lightweight parts that are inexpensive. Not just mediocre substitutes. The wheelset seems like a good start.

You say you will need a new stem soon. The Ritchey WCS 4-Axis is almost the lightest stem on the market, including carbon. And it can be had for about $75 if you try. Ritchey WCS aluminum stuff is solid and quite lightweight. The carbon stuff gets too expensive and saves you little if any weight for big bucks.

Thompson seatposts, especially the Masterpiece, are light too, and will last you a lifetime. There are some super lightweight saddles out there like the Selle Italia SLR at 135g but $150. You might be able to find a cheap one on Ebay that didn't fit someone. On the other hand, the Forte SLX is 210g and $40. Both are most likely much lighter than what you have. Comfort is a personal thing.

At your size, I wouldn't buy handlebars, stem or seatpost in carbon. Stick with aluminum and live long enough to upgrade everything with all of your teeth intact. Many road bike parts have weight limits, so make sure you check to see if what you're buying will support you. Much of the ultralight stuff is meant for racing, and most of them are welterweights or thereabouts.

Tires are a decent place to shed some weight too, but I'd run your current tires into the ground first. Conti 4000S are a good balance of weight, performance and longevity.

If you did all of the above, you might save a pound or so...