If you are in good shape, then altitude won't bother you. You'll just ride slower than you would at sea level for the same effort. How much slower will depend on the altitude, and how steep the grade is.

A few weeks ago I made my annual 12 mile ride from Mono Lake up some 3000 feet to Tioga Pass, in Yosemite. At almost 10000 feet above sea level, I won't say reaching the pass was an easy ride. But it wasn't any harder than any other 12 mile, 3000 ft climb.