You might need a longer recovery period with longer rides. I would think that need for recovery is more closely related to the intensity of the ride than length. Everyone is different though in their need for recovery. I recovery time were plotted on a chart we would see the familiar bell curve with a large spread in times. It is certain that as fitness improves, recovery is sped up as the body adapts to the extra work.
Last week-end I rode two approximately 40 mile days in a row, with another Bike Forums member, with loaded panniers, to a beautiful state park in Massachusetts. I could ride a somewhat longer distance at this point. I enjoyed this tour so much there are now plans afoot for other similar tours this summer.
For myself, if I've done a longer ride, (50 to 60 miles for me), next day I'll do a slow lazy ride of 10 miles through the local state park which is nicely landscaped with many flowers. This works for me and you will find what works for you and leaves you satisfied and healthy.