Generally speaking, not a good practice.
Storing non-food items in a refrigerator/freezer shared by employees for food storage may subject the employer to OSHA penalties (depending on whether the workplace is subject to federal or state OSHA compliance, and other mitigating factors).
It may not do you or your co-workers any harm. But it can be costly to an employer. Even if it doesn't result in monetary penalties it can cost them in terms of time and effort wasted on explaining why an employee's soiled clothing was stored alongside food. More costly if the employer or representative must appear before the OSHA area director or a federal magistrate to defend themselves and negotiate penalties, especially if they have to hire an attorney, professional safety and health consultant, etc.
There may also be local regulations involved, even if the workplace and/or situation isn't subject to state or federal OSHA.
To prove a hazard, an inspector might swab or test an area to determine whether any biological hazards existed, such as hepatitis, e-coli, etc., from soiled clothing. And the inspector might actually go to this trouble. They're spending taxpayer money, so it depends on the agency's budget and priorities.
It won't cost you anything, unless the employer decides you were responsible for causing them a lot of trouble. In which case, they can't legally fire you for something related to OSHA activity unless it violated specific workplace guidelines that were already in place. Or they might just make up something to retaliate, such as gigging you for being late to work. It happens.
Anyway, I'd suggest a mesh bag near a ventilated area or bringing your own small fan to air dry the clothing between rides.
(Yup, I was an OSHA inspector.)