The problem with anything loose will greatly increase your aero drag and the amount of work you do.

I used to pull away from another cyclists I ride with coasting downhill in the winter because he wore a non-cycling jacket that wasn't tight. The same rider and I in the summer would coast at the same speed. So the difference is significant, especially when speed increases.

Cycling tights not only make you faster but it makes for much greater work going the same speed.