Actually you might want a more bulky shoe with them. My Pearl Izumi MTB shoes fit really well inside. I'm thinking a shoe with a really low profile would allow for too much wiggle room.
May I suggest puting your foot inside a ziplock baggie before puing your chucks on and doubling up on socks?