According to Strava, my rides on my 17lb carbon bike and my 21lb steel bike are pretty similar. My avg speed on the same rides is always around the same regardless of bike. And yesterday I did I ride that I have been doing 2-3 times a week for 3 years and I still managed to hit some PRs riding the heavy steel bike.

The characteristics of the two bikes are very different and weight isn't the most important factor. But with that said...$500 to loose 1 lb sounds like a decent upgrade.