So I've been thinking this would go away the more I ride and my body adjusts but it hasn't.

While riding last night it was the worst, so stiff. I did about 21 miles. Mostly climbing and descending. Well the pain eventually got so bad that I couldn't even look behind me for cars.

What could be the cause. My saddle is the correct height, could it be my bars or stem?