I've been plagued by this for quite a few years, across several bikes and across road and TT position. In my case it's always down hill and for me I think 3 components are present when it's likely to happen. There is a road surface component, certain types of regular & higher frequency bumpiness, think rumble strips or roughed up chip seal. There is a visual component. On a wide open descent I feel more prone to wobble than on a tree lined, visually closed in road. And there is a mental component. Anticipating the onset of a wobble on every down hill I'm certain leads to more tension.

Two simple things that really helped me (other than just avoiding some routes, which I do). First is having my head up on descents. Second to a much lesser degree is the standard knees on the top tube. I would also agree that at times I feel more stable with one foot down, or even pedaling.

On the positive side. My current frame, a Felt FR1, is the most stable I've felt in years. It also happens to have a very stiff front end in a sprint, coincidence or related? I'm using HED Jet wheels with 25's. Together a soft but fast feel, a lot of road roughness doesn't make it to the bars. On this bike I can actually tuck on down hills again! Something I have not had the confidence to do in a long time.