Are they metal? (You tell me)

Does your sweat contain water and salt? (The answer is yes)

Then yes. Sweat CAN rust the metal frames. Plastic frames? No problem. Just wipe them down so you don't get a nasty sweaty residue build up because...ew.

Metal...they are probably painted and/or clear coated or are stainless steel. So they should be fine if you sweat on them then wipe them off when you're done and clean them regularly. Wipe them down when you're done riding for the same reason as plastic, and by doing so you remove the moisture and salt that will speed up corrosion.