I have occasional neck and shoulder fatigue when I slack off at the gym.

1 to 2 light sessions per week works for me.

Also, try magnesium oil (a solution of magnesium chloride in water) to spray on the shoulders after the ride. It melts tension away.