I don't have any personal experience with this, but you can try getting a waterproof phone mount. If it's airtight, the heat from phone should keep it warmer than having it out in the cold wind. Sorta like a rain jacket.