One solution is to only wear your bike shoes. I find that I can get by with only my Sidi Giaus if I do not plan any serious hiking. Serious meaning more than a few miles or difficult terrain.

There are a lot of good choices for off bike shoes if you do want/need them. I have used minimalist running shoes and also like my Cushe "slippers". I think the Cushe slippers are my current favorite. They are very light and don't mind being smashed in a pannier, and I can hike in them.