This is one reason I don't mind riding alone or with a very small group of friends. 1.5 hours for lunch, to me is overkill. for lunch.... 30 minutes max really.
Last week on a club ride, it was 57 miles with 3,500 feet elevation gain with pizza stop in the middle for lunch. I left about 2 hours before them, knowing I am slower on the hills and enjoy riding in new places with myself (or a small group of friends). I did not stop for an hour lunch like the club did. I stopped at a grocery store, grabbed fruit and two cliff bars and left. When I take long breaks, my legs hurt and its rough to start up again.