A while back, I was noticing how hard it was to ride into the wind, and then my speedometer lost the connection, and all the sudden it got easier. The thing was, I just couldnt happily ride into a headwind at 10 mph, but if I didn't know how slow I was going, it was okay.

My theory: I think your stoker is a weaker rider than you, and instead of relaxing and and using a compromise pace, you're trying to maintain your single-bike pace and working too hard to do it. Try turning your speedometer off for a few rides and see if it's any more fun.