You've opened quite the can of worms here. Some people swear by skinnier tires with high pressure because they show lower rolling resistance on the rollers that have traditionally been used to measure it. The catch is, unless you're riding on a perfectly smooth surface, those tires aren't optimal. Handling is another matter altogether, but larger tires tend to make steering more stable.

I can offer anecdotal evidence that moderately, or slightly wider tires (25 - 32mm) roll marvelously when optimally inflated. I've run regular Pasela's in 32 (85 and 90 psi), and grand Bois in 26's (95 and 100 psi), and they give away nothing in terms of rolling resistance to skinnier tires I've run that are run at higher pressure. Given what you're saying, I suggest finding a size that gives you handling you like, and then finding a good tire with supple casings and a fast compound on the contact surface in that size.