I carry a little Fuji FinePix model, and have some freinds that really like their little Cannon Sure Shots. Both cameras are small, about the size of a wallet, and take great simple-pictures (nothing fancy, mostly point and shoot), and are innexpensive enough so that I don't worry as much about dinging the camera while riding with it.
I prefer to carry mine in a tiny, water-proof, shoulder-strap type camera bag. I set the shoulder-strap length such that the bag hangs right above my belt-line when in the front, and while I'm riding sling it up as high as it will go on my back and behind the opposite shoulder from the shoulder strap, just behind my arm-pit. In that position, it does not slip down and get into my way while riding, and is still handy for a taking fast pictures of wildlife and such. Some situations don't give you enough time to stop and fish around in your seat bag for the camera.