I feel the same way, to the tune of 1-2 mph. Around here 90 % of the roads I ride on are very course chip and seal. There are a couple of newly paved roads I can get on that are like Nirvana and I see an immediate increase in speed. It happens, whether or not it is psychological remains a mystery to me.