I'm assuming you're wearing baggy style shorts and not lycra shorts?
If not then my recommendation was going to be bibs over shorts all the time.
Good shorts are worth investing a bit more for, you can get very good bibs in the $50ish range.
I think Performance Elite/Ultra stuff is solid. I like Hammer Nutrition clothing for the price who at least used to be made by Voler (used their top of the line chamois for a long time).
Check PBK (probikekit) also as some of their closeouts and team clothing deals can be exceptional.