I've found the same thing. I think it's because I don't get to ride at my natural cadence.
Last Thursday I rode 36 miles with a group of slower riders, all of them only occasional cyclists. There were 11 stops on the ride, and it took from 9:30 am to 5:00 pm. Ride time was over three hours. I was exhausted by the end of the day. I've done centuries in similar time and felt just fine afterwards.
The same thing happens when I'm walking. When I have to toddle along with my parents, I find I'm worn out in no time.
EDIT: Stapfam may have a point too. Last Thursday my average HR on the ride was in zone 1, and my max in zone 3. Never really got warmed up.