more than likely any clothing that flaps around creates drag.
when it comes to shorts/pants, if your legs are sliding around inside the clothing, that creates resistance.
its much more difficult to ride 60 miles in a pair of knicker shorts that constantly rub against your knee/leg. a lot of effort is used in that articulated area to over come the clothing.

I like baggies too, but mostly wear them in town if I'm running errands.