Hal is onto the correct answer. I once again defer to the highly regarded Bill at Torelli, for his explanation of why there is such a thing as too narrow. He's writing about width and pressure, and while this article is about road biek tires, it holds true for skinnies on your MTB:
Tire pressure. In the last decade it has become an article of faith that the more pressure one puts in a tire, the better (that is, faster) the bike will ride. One brand of tires actually requires 140 psi. This kind of tire pressure makes the bike ride like junk. With super high tire pressures the bike bounces, transmits road shock, fatigues the rider, causes the premature failure of bike frames and components and makes the tires more prone to flats. Running the tires at 105 - 115 psi will yield a bike that corners better and gives a far nicer ride. Try it. It will still go fast.
Tire Width. There is a view that a 20 mm width tire is faster than a 23 by virtue of its smaller cross section and lighter weight. Interestingly enough, this is not true. The people making the Torelli tires had noticed that the pro teams that they sponsored asked for 23s because they felt they were faster. When they investigated and did the testing, they found that the riders were correct. Let me explain.
Let's assume a 200 pound rider and bike unit. Let's also assume that the weight is distributed half over each wheel. That means that each wheel is supporting 100 pounds. Now, with a pressure of 100 pounds per square inch, the contact patch is one square inch. This is true no matter how fat the tire.
What changes when the tire gets fatter is the shape of the contact patch. With a 20, the contact patch is a long oval. With the fatter tire, the contact patch gets shorter and wider.
When a rider is using a skinnier tire, the long contact patch means he is flexing a wider arc of the tire casing, flexing more of the tire, causing more wasted energy from the internal friction of the tire and tube. The rider with the fatter tire is flexing fewer cords at a time.
There is clearly an optimum size, and the fact that racing tubulars are around 22 should keep us from getting super wide tires looking for yet more speed. Other losses probably kick in as the tire gets still fatter. For me, the bike feels like it doesn't have any snap or jump when we stray from the optimum which I believe to be in that 22-23 mm range.
Some have suggested that the skinnier tires make up for their losses because of their lower aerodynamic drag. This could be true for the solo time-trialist, I'm not sure. For the pack rider, it clearly is not an important consideration.
Food for thought.