Perhaps this has already been said in a different way, but a significant crosswind combined with your forward motion creates more turbulence than you would experience from the same forward motion through otherwise still air. And turbulence is bad, of course. This is the reason for so much aerodynamic design - not just getting a low drag profile, but also to minimize turbulence. (Think about those funny time trial helmets). it's also why you can go a little faster with people drafting you than you can solo for the same effort - the people behind you reduce the turbulence.
Citation to the assertion that there is an aerodynamic advantage when people are drafting behind you:
Training and Racing With a Power Meter Journal: Does drafting benefit the leading rider?