Your speed is somewhat a function of what your goals are. If your goal is to get from "A to B" and feel good while doing so, and to feel good after arriving, a comfortable speed is fine. I know some guys who ONLY ride at maximum speed...all they ever talk about is "suffering" and "tolerating pain" and the possibility of knee surgery.
There is also a "tortise and hare" thing going on. I will sometimes be doing slow laps around the "sorta" track at Rice University. I've been doing laps for thirty minutes. A young guy drives up with a bike on top of his car. He riaces around the track for fifteen minutes at amazing speeds. He gets back in his car and leaves. I continue doing my slow laps for another thirty minutes or so. Then, I go ride for an hour or so around the neighborhood.
My long, slow rides meet my goals, which are based more on relaxation and enjoyment. His fifteen minutes of speed meet his goal, which is still something I have not figured out. What IS the point of driving across town to do a fifteen minute sprint and then drive back across town?
I see the same thing on bike trails. The "speedsters" are riding fast, but they seldom are on the trail long. They want a fifteen minute fast ride. I want a two hour slow ride...different strokes for different folks. Let the "fast" guys do their thing. And, just do your thing.