Yes, a crosswind does slow you down. I often ride on a lake dam road which runs east and west. Wind here is usually from the north or south, so the dam rides usually have a crosswind. The harder the wind blows, the slower my average speed will end up being.
Different roads behave very differently with winds from different directions, too. On a repeating road route loop near my home, I have the best speed average when there is no wind, but a north wind only slows the average a small amount. A south wind, on the other hand, makes much more difference, decreasing my speed average quite a bit.