I wouldn't worry about what others think. You're happy with the purchase, go ride and have fun. FWIW, you're focusing on the part of the bike that may have some significant impact with upgrades. I can't find the reference (actually quite few on average TdF speeds), but I do recall an article on average increases in TdF speed from the end of the steel bike era to the present. 3 kph or so. The author attributed it primarily to improvements in wheel technology with frame stiffness (carbon engineering) - but not frame weight primarily, a secondary contributor. I'll keep looking.

Enjoy the wheels.