loose clothing up top is a drag impact, but for me it's mostly a noise annoyance
loose clothing down below is asking for major chafe.
dr_lha has it: they are designed that way because that's the most efficient/appropriate apparel for the activity.
(point of reference: I mowed my father-in-law's lawn Sunday wearing my 'kit' ... once you get comfortable, it's all good)