Altitude may or may not slow you down much. I don't think we adjusted our distances much on our recent tour. We definitely felt it though.

At above 4000' we just felt like we were a bit out of shape until we were at that altitude most of the time for a few days. The same thing happened as we went up to 7000-8000'. When we finally climbed over Hoosier pass (11500') we felt it again, but climbed pretty well. We were all OK on the climb, but one of us felt sick the next day.