When you arrive to work, Do you hang out like in the cafeteria or chill in your office for a while before changing into working clothes?
Are the shorts too tight?
The subject is very sensitive; Could there be other issues at your place and the employees finally found an excuse to complain?