I use a pretty cheap ding-ing bell that I can hit a few times. All but headphone zombies seem to move on the MUP... Although I've noticed small children usually move the wrong way, I always slow down and wait for them to figure it out if I have to warn a family, parents seem to not keep an eye on their kids and let them wander into bicycle traffic around here.
A whistle seems to be an awkward thing to have to reach and grab and I'm not sure people are as well 'trained' to react to a whistle and assume it's a bike trying to pass.

A bell seems to work well enough in Japan: http://www.youtube.com/watch?v=O_MphtzCOEc