Everyone is talking about drag and time spent in headwinds, and while that's all true, this discussion isn't the full story unless we're assuming someone is holding the same power throughout their ride, and has the same drag coefficient (so in the drops the whole time with the same arm angle or whatever).

Like I said before, a lot of our anecdotal evidence about winds is a mental thing. I definitely push myself more throughout a ride when I start with a tailwind, and there's the added bonus of always going further than I intended when I set out.