The SPD shoes I wear also have a hard plasticky material on the sole where you would think they are rubber, so they can skate around on slick tile, wood or stone. Also if you happen to step on something right on the cleat like a small pebble on afore mentioned surfaces it can make it slide out suddenly too, softer stuff might get wedged in the cleat and make clipping in hard. Walking around a lot might wear the cleat and sole pretty fast, tear up your socks or feet too.

As others have already mentioned, just leave your work shoes at work, alternately get a back pack, rear rack and a bag or panniers to carry them.