I think it's the geometry of your frame and/or fork, not lack of stiffness in the headset, that makes your bike respond so quickly.
A bike with a shallow headtube angle and a lot of trail will be slow to respond to steering inputs and more stable in a straight line. Touring bikes are designed this way. Racing (particularly criterium racing) bikes have steeper headtube angles and less trail and are made to be super responsive but less straight line stable.
You do not want stiffness or drag in the headset under any circumstances. You don't want free play but it should turn easily and smoothly.