I can be pretty slow... and do get passed every once in a while... although I also frequently pass slower people.
I have pipe dreams of going a bit faster
However, I really ride to get from point A to B, and don't choose to get 100% energy depletion in the process.
It is all too easy to drop from whizzing along to plodding along.
As I have increased my riding in the last couple of years, I have noted that strength and endurance seem to be improving faster than the actual average speed.
I've never been in the super-heavy category, but I'm working on loosing a few pounds this summer (and hopefully I can keep it off next winter). I know there is a lot of debate about the effect of weight, but car go does slow me down, so I would imagine carrying 10 or 20 pounds of excess butter around with me will also slow me down.