My main challenge comes before the actual riding- its dedicating time to be away from work and life's obligations. And that's just for shorter multiday tours.
While actually riding?...I guess its rain. Not really a challenge, just annoying to pack up a wet tent and stuff. Im really picking nits when that is the biggest challenge, though.