Nothing wrong with an easy ride now and then. You can always add some solo miles to the front or the back of the ride, especially if you meet him somewhere. Or just go out the next day and clobber yourself.
I found the truth in this just yesterday. My wife joined up with me on mile 44 of my ride (I was doing mup loops), and we did eight or nine miles together at half speed. Not that I could have bragged about the time anyway. But after we were done with that I was well rested and finished up at a stronger pace than at the beginning, coincidentally my first 65-mile ride. It's all good.