Get some other kind of pedal/cleat/shoe system and be done with it. Probably Shimano SPD (the metal kind). However little you think you're walking in those shoes, it's too much. Case closed, it's just that simple.