I'm pretty slow going to work, to stay cool. Then I ride fast going home, for fun. Also, it's net downhill going home.
My speed tends to depend on my attention span. If I start daydreaming, then I slow down.
I'm also not that fast on leisure rides, and am not even equipped with a fast bike these days.