"Much faster"? What speeds are you talking about?

Since racers don't use toe-clips with sneakers, it could be something funny with your technique or a change in position/fitting.

For most riders, the speed difference is likely to be fairly small (not "much faster") anyway.