Think of your bike as a tool - is it adequate for your needs? Would a new bike enable you to go further, more comfortably, feel more confident on the downhills?

Go to your LBS and test ride the bikes in your post, and others if you can. Make it a good test ride - not just a few laps around the parking lot. Only then can you tell the difference between your current ride and a Specialized or Cervelo. If you perceive no difference, then save your money.

No need to spend $3K either, as most of the main brands (Specialized, Trek, Cannondale, Fuji et al) will have bikes in the $1-$2K range that should be an improvement over your current ride. A test ride of the same model but at different price points will help you decide which price point is sufficient for your needs. When you do dollar up - keep your old bike and use it for bad weather days or going to the pub.

My own experience was that - yes it was worth it to have a bike (a Specialized Roubaix with Shimano 105 in this case) that was stiffer under acceleration, smoother over rough roads, more comfortable on the long haul and tracked better on the descents. Having a bike that fit the type of riding I aspired to allowed me to go out and meet those aspirations.