Yes, cross winds will slow you down. If you are riding with others in a cross wind, rather than drafting directly behind the rider in front, you want to form an "echelon" in which you are behind the rider but off to the left or right of the rider, depending on if the wind is from the left or right.

Actually, cross winds can be more tricky than headwinds, since the wind will move you off course. This is especially true if the wind is gusting. Even more true of you have deep rims on the front.

Back when they started using solid disk wheel, they had them front and rear. They quickly found that disks on the front were a bad idea when the first wind gust blew them across the road.