It's the Ulnar nerve (most likely), and since it has to go over the first thoracic vertebra, craning your neck tightens the muscles around it, thus causing numbness.
It could also be from riding on rough road or gripping the handlebars too tight (or any combination of the 3), as those can cause trauma and stress to the nerve as well.