if you've only just started the longer distances then its simply that those muscles which have to deal with weight bearing like holding your head up while you ride are tired.
if you just bear it out those muscles will get stronger and you will be able to stay in the aero position for longer instead of having to adapt your bike to give you a more upright position.
i had it when i first started road riding and the best way to deal with it is to just bear it. a bit like breaking in a brooks saddle, you're breaking in your back muscles to hold up your head for long periods of time.
if you ride long distances regularly you might notice (after a while) your back, especially lower back muscles getting stronger and getting bigger.