geometry has a huge effect - steep headtube and/or not enough trail will make a bike spooky at downhill upright speeds. Hence track bikes, many TT machines have a hard time on long descents at steady speeds. It's usually where high speeds remain constant and the bike angle is kept upright.
I have an older Limongi custom frame designed for very quick response in crit racing - very scary above 35 mph on a steady downhill - but extremely nice at 35 mph in a fast, flatish crit.
Even though a headset might not seem 'tight', backing off the downpressure of the stem might be enough to help eliminate some wooble. Too loose of a headset can have the same effect.
Stiff arming the bike will accentuate wobble - the bike front needs to 'hunt' without resistance. Also Stiff arming causes the opposite of what you need by moving your weight distribution rearward. Get more weight onto the front end for more stability.
check the wheel hubs...
If you feel the wobble coming on, besides the knee/thigh clamp thing to the top tube; you can also (if you have enough roadway width) go into a very shallow weave. Putting the bike over just a wee bit will often stop the wobble from increasing.