The longer the distance you ride, the better your fit needs to be. If you're just riding down the block to the store, you can get away with riding your bicycle set up in any old way. But the more time you spend on the bicycle, and especially the more revolutions your legs do, the closer to perfect you need your setup to be. If your bicycle isn't set up well, you might be less efficient than you could be, and you risk injury.
That said, fitness plays a very important part in riding longer distances too.