Speaking anecdotally, I find 23c's to be really hard, hand numbing, and buzzy (at ~110-120 psi). on one bike, I run 700X28, and appreciate them quite a bit - at 90 psi, they're more comfy and rugged than the 23's, but are still pretty hard. another bike I've got runs 27X32's, and that one is ideal, IMO. The difference in drag is negligible between 28 and 32, and you can run them at 75-80 psi, get plenty of cushioning, and not have them wear you out. I've the feeling that most of my bikes from here on out are going to get tires somewhere around 28-32c range. Another thing to contend with is that the roads here are badly potholed and cracked. With 23's, the cracks will railroad your wheels into them, causing you to biff it. With 32's, I just roll right over them.

IME, I would say that for anything but smooth road riding (i.e. anything with any gravel, chip seal, cracks, etc), stick with 28's or higher.