IMHO, 105 or ultega isn't so much the question - it is the whole component configuration and bike fit.
Based upon the type of riding OP indicates he does, a slightly relaxed frame geometry could made more sense. Good fit is a key component. Ultega RD and shifter with a clunky wheel set and generic brakes probably.isn't any advantage over 105 set up with comparable components and.wheels.