I don't carry an MP3 player or anything else with me on rides so I have thinking time. You would think that a wind coming from your 9:00 or 3:00 position would have no effect on your speed or effort but it sure doesn't feel that way when you get hit by a gust from the side.
Just playing with even numbers, say you're riding at 20 mph and there is a 10 mph wind at your 3:00. The felt wind will be 22.3 mph coming from your 1:00. Keeping numbers round, instead of having to bore a hole in the air at 30 feet per second, the hole is bored at 33.5 feet per second. Considering that the energy cost of moving air out of the way increases as to the square of your velocity, the side wind cost you 25% more effort to maintain speed. Things get worse rapidly as the wind shifts to your front but the wind probably has to be from 4:00 or further back before it doesn't cost you.
If there are any errors in my reasoning, I'd like to know it.
I live in northern Florida where there are relatively few days through the winter where it is too nasty to ride but there sure is a lot of air in a real hurry to be somewhere else.