I stay clipped on both, as I try to use the upstroke as much as possible, especially on steep hills. I have a theory it's better for my knees to work a wider set of muscles/ligaments etc.

On my SPDs I can unclip quite fast, so have never really had this fear.

SharpT