You lose a lot of control in my opinion. I've rode to the store just as your suggesting, without my clipless shoes and it just doesn't feel right.