Like you, I keep track of distance and I don't keep much track of speed ... but I'm starting to keep track of the speed because I want to increase my speed ... so I guess that the first step to increasing speed is knowing what my speed actually is!
I don't know if you have a log, but I've been logging my distance for the past 15+ years ... it is quite an extensive log ... but not a word in it about how fast any of the rides took me!!