I've found that there's a pretty large variation from brand to brand. I have some Pearl Izumi large bibs that I can swim around in, and some Sugoi bibs, also large, that can be challenging to squeeze into.
Although it may not be relevant to your situation, better shorts usually have more panels, and the panels grip you a little more firmly.