At either the handlebar clamp or the steerer clamp, one cannot set bolt torque by gauging clamp tightness. There needs to be a gap in the clamp at each bolt, and the bolts should be torqued to specification. A broken bolt usually indicates over-torque. BTW, which bolt broke -- at the steerer or the handlebar?

As for the stem spinning on the steerer under force, I advocate a tightness similar to what you describe -- provided bolt torque is correct. That is: the stem will not spin in use, but will yield to moderate impact.