Many good places...Front Range is a good hub to ride from, and even Manitou Springs has loads to offer in terms of a home base.
But, I think Summit County would be a great home base, too...Hoosier Pass within easy reach, Vail Pass too, even Loveland Pass and more (Fremont, Tennessee, etc...). To boot it's a very tourist friendly area with great food, shopping and spas.
Growing up in Buena Vista I think BV and Salida would also be great places to base out of for 4 days...especially Salida, but it's a bit more removed there, and some people say it's an acquired taste. Nice thing about BV is you can hit Independence Pass with ease, and you can also tap Poncha Pass and Monarch Pass, and if you're ok with out-&-backs Cottonwood Pass is one of the sweetest views evar.
Honestly, it's really hard to go wrong in this state for good riding. Better yet is you'll always be able to find a quaint, mountain town to base out of that has good food and comfy beds.
Like others have offered, pm me if you want more.
Enjoy!