There is far more to steering response then one simple number, as in trail. This is why after 150 years of bike making we're still talking about what makes a good handling bike. here are some of the factors- steering axis angle, rake/offset, tire diameter (and these three determine trail), front center VS rear center lengths, Bb height, wheel/tire inertia/mass, stem length, bar width, center of mass of the bike/rider system and I've probably left out other factors.
I've looked at this since 1979 and still feel that my understanding is incomplete. Part of the challenge is that as many people write about steering design there are opinions as to what kind of result is best. Right now a low trail geometry is in fashion with some riders. If you look at what racers use (and we all know that what racers do is really the only guide that counts
) a long trail design is better. And of course few mention a head angle or wheel mass when discussing this.
So I suggest that you first get schooled on the terms and relationships. Then at some point you have to start riding the various designs to develop your own opinions as to what design you find handles best. Some of the publications and information sources I've used are Bicycling Science by David Gordon Wilson (get the third edition which has Jim Papadopoulos contributing). The mentioned Moulton blog. Bicycle Quarterly and Jan Heine. These are a good start. But beware of the opinion that gets published as fact.
Look up the old Scientific American article "the un rideable bicycle" (I might have the title wrong). It was a bunch of smart guys trying to design a bike which couldn't be ridden or steered. They found that pretty much any design they tried was able to be ridden, some were far more stable then others though.
As to other aspects of frame/bike design- I agree that the current fashion of really short chain stays is miss placed. My take on this is because stay length is easily measured (and BTW trail isn't easy to accurately measure, most claims rely on manufactures supplied angle and rake dimensions) stay length is used by the marketing departments to their advantage. Longer stays result in smoother chain runs, more cog/ring combos being useable, more stable bike handling in general and a host of other real world advantages.
Wheelbase is only a result of all the other factors that go into a bike design. Like the riders fit/contact points, the steering geometry, the stay length and possible need for clearance with feet/bags, the BB height and more. So again, since measuring wheelbase is easy it's used by marketing to their advantage.
Top tubes only hold the frame together, provide a cockpit for rider fit, and influence frame stiffness (especially when heavily loaded).
Do you see a pattern here. The easy stuff to talk about is claimed to be important and the hard to measure stuff is subject to opinions claimed as facts.
Lastly the bike's design to be best at the type of riding one does off road is vastly different then for on road. How the rider positions themselves, how they move and 'work' the bike, how the bike's relationship with the ground changes is very different off road then on. How the bike handles added weight or needed accessories isn't a design factor for an off road performance bike. So it's no surprise that your experience is what it is. Andy.