Doubt you'll hurt the pedals but slipping off the pedal with your street shoes would hurt- a lot. The smaller surface area and the less grippy interface of a clipless pedal can never equal a real platform for security on street shoes so think about that. 99% of the time the difference won't matter but it's that 1% that might earn you a darwin award.