Sure. I know what you mean....
I don't pay a lot of attention to speed, really. I do pay attention to how much effort I'm putting into those pedals. Under favorable conditions (level, tailwind, etc) I'll fly along at a good pace. But I won't torture myself to keep it up in a headwind, for instance. I'm content with riding not much faster than walking when the wind or the grade are working against me.
I want exercise. (And I get it) But I don't want to be obsessive about it.
Like some have said, "Stop and smell the roses". It's good for you.