One of the first things I do upon getting back to the house is rinse my helmet off under a stream of water in my laundry room sink. It probably matters not whether it's hot or cold, but the HOT is easier for me to get to when holding my helmet in the other hand.

If I decide to go riding before it dries, so what! It is going to be sweaty soon anyhow. And the water remaining just adds to the cooling effect for a while.

Rinsing it off does keep it from getting all sticky, nasty and smelly.