Not that much difference from this average rider. My last bike was a Scott Speedster S20, a midrange Aluminum bike. My current bike is a Guru Evolo-R, a higher end handbuilt carbon frame/fork. I can't tell much difference (some, not much) between them (very similar geometry between these 2), as least not to justify the 4x price difference had I of paid retail on either. Another reason why I'm cheap and buy used.
If I'm going to drop $6k+ on a bike, it better be a magical experience over the $1500 bike.