I was focused on my average speed like a Prius driver fusses over their average MPG. I felt the same as you, and found myself stuck in a game of maintaining some number, and fussing over a 0.1 drop. My solution was to put the cyclocomputer away and only take it out on rec rides to track miles. I'm much happier, and my travel time is about the same.

My average now is "fast enough"