Lower or higher gusts are always present. When come from the front, they produce short, intermittent surges of power that muscles don't like at all (tail wind generates the opposite, which is perceived as refreshing for muscles). Breathing is always a little more difficult with front wind (I never could explain this).
Then: speed is lower, wind noise is higher - these contribute to overall decrease in comfort, which psychologically is perceived as making a bigger effort.