Definitely the rim shape is a major factor, even on older, not-so-tall rims, and especially with the more-pointed rim profiles.

My old Kestrel was terrible in even mild crosswinds until I tried swapping front wheels with a fellow rider while out riding. The box-section rim fully cured the problem that the low-profile 1980's "aero" rim had caused.

There can be an abrupt change in flow pattern as a certain angle of attack of the passing air is exceeded, as a largely laminar airflow pattern suddenly transitions to a turbulent airflow. The effect on steering when this occurs can be dramatic and startling. I do suspect that pointed rim profiles encourage this sort of abrupt transitioning.
My worst encounter was while riding a loaned pair of Mavic Cosmic Carbone aero wheels, which while giving the bike a definitely faster feel did also cause me to think at one point that someone had actually grabbed and pulled at my handlebar!

Since I am not doing any road or crit races these days, I want nothing to do with things like twitchy rims or carbon braking surfaces, things which detract from the pleasures of cycling.
Nor do I want to smell other rider's cooking brake pads nor see them getting blown toward traffic or veering toward the roadside ditches.