Great question, because my perception of the urban dwellers and hipsters saying "you need only one" is, they must not be going very far.
I live in a river valley on a hill, so I have to climb a lot. A typical after-work training ride for me is about 15 miles. When I go longer, it's 20-25 miles before I need a break or I'm done. If it's somewhere flat, sure, 40 miles is no problem. It's actually not so much the climbing or fatigue, it's also more mentally tiring on a fixed gear.