Well, TA this has certainly taken on a life of its' own. Your original post certainly sounded to me like you were deriding the slow folks because you could ride faster. If that wasn't your intent then we certainly have gotten ourselves crossed up in the process of swapping written words. You are right it is tough to communicate meaningfully in this kind of environment.
As for my comments about it being all in a person's head, let me put that thought into a different context. Many of us have had jobs where there are busy days and very slow days. It is a common for a person to feel tireder at the end of a slow day than at the end of a very busy day even though they expended much less energy. Difference? Mental attitude. There is something I call the Boredom Factor that makes a person feel tireder at the end of a slow day than if they had been busy all day.
Put yet another way; the perception of time and energy is just about as important as the actual numbers measuring those factors. Think the old saw about 30 minutes of sex going much faster than 30 minutes in the dentist's chair; and leaving a person much less tired, maybe even invigorated. Difference? The mind?
So, although there may be some physical factors influencing how tired a person is after riding slower than normal, the primary cause is probably mental attitude.