I use polycotton hiking pants which dry in about 10 mins. In winter I use pertex overpants (similar to Montaine featherlite). These can fend of pretty heavy cold rain and dry within minutes. If you do get a bit damp underneath, the polycotton copes well.
Pertex pants pack down really small so carrying them is no problem.