Yeah, the trust thing is a big deal when drafting or riding close side by side. And after a crash or three I'd bet most of us tighten up our trust standards.
Of the road group I ride with occasionally there's only one guy I trust enough that I'll draft him even at night on a narrow twisting path, having no idea what's ahead but his butt. He's that good.
I try to return the favor but I don't have his bike handling skills on fast curves and twisties. I always lose a split second, a gap opens and I'm dropped. But I'm okay on straights and rollers so I'll pull for a mile on those segments. Good workout too.
A few other folks are regular leaders on casual group rides too so they're pretty predictable, although I don't trust them quite enough to just lock my gaze on the wheel and follow. I'll peek around as we'd normally do.
The other guys... some are pretty fast but they'll run you right into traffic cones, potholes or debris without warning. Some wobble around or shift lines randomly without seeimng to be responding to actual road hazards.
On the plus side, riding with folks I don't really trust made me stronger because I'd drop out of the draft and work harder to keep up. Sorta Cat 6-ish. Not quite the experience I was expecting from a moderately fast club ride, but, hey ... lemonade from lemons and all that.
Best drafting experience I had last year was when one of the regulars had to concede to age and started riding an e-assist road bike. He's very smooth so I'd get in his wheel on climbs and rollers. Lotta fun. Closest thing I've gotten to pacing off a Derny.