I always wear them. I use SPD shoes on my commuter (which has XTR SPD pedals), these shoes are also good for walking around. On my road bike I use Shimano carbon-fiber shoes, they are more comfortable than the Sidi and Carnac shoes i used to wear. I prefer Look pedals on my road bike, not exactly good for walking, but great for power transmission.
Unfortunately, I found to get my best pedals stroke using quill pedals with clips and straps. I have not been able to duplicate the the measurements on clipless pedals and shoes. At least with clips and straps you can get away with wearing most types of shoes.