If you're concerned about weight, most people seem to find the extra cost of Ultegra is worth it, but not DuraAce. I tried a bunch of bikes before I bought mine, and I found Ultegra to be smoother than 105. My bike is mostly Ultegra (crank included), except 105 FD & cassette. Brakes are excellent, as other noted.
Unless you're a top competitor, really it comes down to budget and how it looks & feels to you.