If you can't torque it down enough directly to the handlebar, I'm suspecting a design flaw.

In the way distant past, I used to rock a Minoura SpaceBar for lights and a computer. This was the days before Garmins and phones. I don't remember ever having this issue.