This seems really petty and I bet your supervisor is annoyed at the people who are complaining. However, probably the simplest and most effect way to deal with it is to put a pair of gym shorts on.

It kind of annoys me that these women think they have a right to complain about this as women tend to be the worst offenders in this area. Women's office casual ware has gotten pretty form fitting and revealing in the last several few years too. It is another one of those where do your rights end and my rights begin things.