I have been through similar tests, as my initial bike when I re-entered the hobby was a Giant Rapid 2 (very similar to a Defy frame, slightly longer dimensions, with ALUXX SL frame, Advanced Carbon fork, and Carbon seatpost). I quickly caught the bug, and decided to move to a traditional road bike, and chose a Defy Composite 2. I kept the Rapid, and have ridden the bikes multiple times on the same surfaces, and find the Defy Comp to be smoother, especially on rough pavement, and much more responsive. Both have the same level of bar and stem (Connect Aluminum), same fork, but the Defy Comp has a different carbon seatpost than the Rapid. The Rapid has 25mm slick tires, while the Aluminum Defy 1 you tested, I believe, has 23 mm tires like my Defy Comp. I keep them both inflated to 110 - 115 psi. Wheel sets are comparable. Another key difference is my Defy Comp has a SRAM component group, while the Rapid has Tiagra/flat bar components and an FSA crank.
Maybe you do have a higher comfort level with Aluminum? I love the Rapid, and the flat-bar stance, but I've come to prefer the overall feel of the Defy Comp. Hope this helps!! Either way, both bikes you are considering are Excellent. I'm sold on Giant's bikes!