In general I think that wheel size is much much less important than people think it is. People think that since drastic changes in wheel size result in drastic changes in appearance, it should also indicate drastic changes in rideability. I believe this is a misconception similar to the misconception that small wheels require you to pedal faster. People that understand gearing know that small wheels do not necessarily require you to pedal faster, just that you need to compensate the gearing. Likewise, I believe that small wheels do not drastically affect ride quality as long as they are compensated for elsewhere.

In my opinion the only disadvantage of small diameter wheels that can't be compensated for by other means is the tendency of the wheel to get stuck in cracks. If the diameter is smaller than the cracks then this becomes a problem of safety. If the diameter is bigger then the crack then this becomes a problem of efficiency. Everything else can be compensated with suspension, longer wheel base, etc. For example, look at cars. Despite wheels of around 20 inches in diameter, they seem to manage fine at extremely high speeds on extremely poor roads. They compensate for the small diameter with both a longer and wider wheel base and a sophisticated suspension system.

Thus, neglecting efficiency, the smallest practical wheel size is slightly larger than the biggest crack you plan to encounter. I think six inches would be fine in most situations if you had wide tires, good suspension, and a long wheel base. Of course, these things would make the bike larger and heavier, somewhat negating the advantage of the small wheels.