Really, really, really bad idea, imo. If you are pedaling hard, especially while hammering up a hill, you will certainly pull out. If not, you won't be able to pull out when you need to.

Go with some comfy spd mtb shoes. I could walk around in my tahoes all day long.