Why not put that cog on a little axle, and mount the axle to a clamp on the chainstay? You wouldn't have to worry about it falling out, and you could adjust tension by sliding the clamp forwards and backwards.