Yes the penalty against the wind is bigger than the benefit of the wind at your back. You lose an amount of time going out. Coming back you have less time to make up time since you are traveling faster.