You always get a little zinc fume when brass brazing, especially fillet-brazing where the flame actually points at the rod sometimes. The health effects, according to all sources I could find in my non-expert search, are acute not chronic (i.e. temporary), no permanent damage done except at high concentrations. Re-melting a cold fillet, or sweating apart a joint with large flames, releases a lot more fume than proper brazing where you never get it too hot.

I know of at least one death from zinc fume inhalation, but that was from melting the zinc off a bunch of plumbing parts in a kiln. When he opened the kiln in his small workspace it was filled with a lethal dose.

I have felt a mild form of the short-term effects which are flu-like: nausea, headache, etc. That was from fillet brazing for hours, like making a production run of tandems. Having felt it and not wanting it again, I started wearing a cartridge respirator, with a plastic face shield over that, when doing long stretches. The cartridge to use for metal fumes is HEPA. For most brazing though, it suffices to have good ventilation and keep your head out of the plume. In my non-expert opinion.

Wikipedia article here seems good: https://en.wikipedia.org/wiki/Metal_fume_fever
Note they say that onset of symptoms is delayed, so you can't just braze until you feel symptoms and then take a break. You may not feel it until you get home. That jives with my experience.

There are other metal fumes that are way more dangerous than zinc, like cadmium and hexavalent chromium ("chrome-6"), which definitely kill.