1. Even smaller panniers can get hung up on a rock or branch when navigating singletrack
2. Panniers tend to make an inordinate amount of noise when bumping along a trail
3. Panniers open up the temptation to overpack.
Bikepacking bags certainly have their negatives but I appreciate folks trying other ways. One thing I've observed over the last couple of years in the bikepacking world is the adoption of small racks and small panniers in addition to framebags and other bikepacking bags. I see it as a correction away from the no-pannier ethos that's been pervasive for a while and a recognition that panniers can be useful.