I'm pretty certain both times has happened on rear tires, and one theory is that as I am a lighter rider, I like to run lower pressures on rough roads, as it is less jarring, easier on the bike, it corners better, and puts less stress into the wheelset etc--what might be happening is that even though I have never had a pinch flat (well, maybe once) the rear tire may be flexing outwards more than usual and when I have more things in my pannier when commuting, it may flex out just that bit more and touch parts of my canti brakes or the pads when hitting a larger hole or whatever.

thats my theory anyway, but again, has never really impacted the tire and has never gotten concerning looking or anything past the odd strand appearing that I eventually snip off at some point if I think of it.