I have done several lake and river swims in my cycling shorts on hot days and never had an issue. As for salt, your body naturally sweats it all over anyway on hot days, so it shouldn’t be an issue. Sand might be an issue if you hug the waterline (i.e. body surf). I wouldn’t worry about it. Besides you will get some stares by people wondering about the guy in the old-time swim suit.