Soak for a time in water and Clorox bleach. Or soak in hydrogen peroxide. Or soak in some water and Oxyclean.

Hydrogen Peroxide probably works the best. But bleach is cheaper and always on hand here. I sometimes will have to take the silicone nipples apart on water bottles and soak them good when little bits of mold or mildew start showing. Probably would be better to just regularly soak them before it shows up. But I haven't gotten into that habit.