I find I am having an increasingly hard time making myself take it easy and go slow (as riding into work to minimize the sweat factor or riding with my wife).
It seems that what I perceive as "slow" changes as my level of fitness and ability improve.
Last year, I thought that 10 mph avg was doing great!
This year, I sometimes hit 11 or 12 mph avg and think, "Meh...I'm not working at all."
Next year it may be that 15 or 16 mph feels like I'm standing still...

Am I trying to go faster? In general, No.
That doesn't keep it from happening, though, as my body adapts to cycling.