Get wicking base layer clothing. When I go out in the cold I wear thin wicking base layers and a cotton turtle neck over that. The turtle neck gets drenched but the two, or one if it isn't that cold stay relatively try. By their nature even if the base layers are slightly damp when you take them off they will dry quite quickly if you just drape them over a chair for an hour or so. So if nothing else when you dress again your base layers will be dry.