I don't have that trouble at all. I wear mine up high and the buckle is nowhere near my shoulder. Perhaps you're slightly deformed?

Besides, the newest ones have redesigned buckles.