Sounds like some stoker upper body movement. Maybe try a higher cadence or have your stoker concentrate on a quiet upper body.

A death grip on the bars at the front will also fatigue you and may also introduce weaving - relax as much as you can.