it certainly would be anything but leisurely. i've only done the crest trail once, and it was years ago, but i remember there is a creek bed you ride though towards the end that is very rocky, aside from the hazards bulldozer mentioned. imagine doing williams canyon or elk park on a cross bike. it's perhaps doable, but would also be a great place to ruin a wheelset or a fork. that's sort of a long walk home.
as far as nice long dirt rides, what about riding up rampart range road to palmer lake and down the santa fe trail or up to the summit of almagre? you can take FR 379 all the way there, off of old gold camp/ old stage roads.
how can you live in this town without a mountain bike!?