I'd say it's pretty simple. The faster you go, the more important air resistance becomes. When you can average 20mph for an hour, the clothes you wear make a big difference in air resistance.
Agreed to the first -- that's a fact of moving on the surface of the earth (not in a vacuum).
Meh to the second. If you wear a huge cape I'd see it, but it's really just conjecture. I'll admit straight-up that I don't maintain 20 mph, but even at speed I'm not sure my cotton t-shirts would really "bog me down" out there
.
There's also comfort and sweat management.
I think by this you mean that wearing tights/jerseys are more comfortable, but I actually find this as a more convincing argument for wearing my button cotton shirts and linen.
.