I've been doing a mix of hiking and biking for many years, really never had to force one to take a back seat to the other - just keep it mixed.
I've found biking hills does keep me in pretty decent shape for walking up hills but you are right about those downhills! Similarly, a 3 hour hike is great for your legs but does nothing to keep your butt in shape for 3 hours on a bike seat...
I hiked and biked weekly with a guy for over 20 years who stopped biking when he retired, but did a lot more walking. His ability to hike up hills declined dramatically - could have just been age, but I think the loss of the bicycling cross training had an impact.