A bike with panniers sounds like a decent bike to draft if you're winded. Perhaps he was taking a breather. And as you said, you were gassing it a bit. Or maybe you hurt his little feelings by saying hello! Hehe. A lot of bikers out here are like that. I say hello and they say nothing back. As if a friendly gesture is suddenly a challenge because we're both on bikes... Weird.

Personally I don't mind if someone drafts me. I always try to push myself on my commutes, so being able to compare my fitness level to someone else's on my daily ride is kinda cool.

One time on my way home I had passed a profi looking guy on a souped up carbon cannondale with R-Sys wheels whilst on my wifes 2007 Trek 5000 with standard Bontrager race-lite wheels. He didn't like that so much so he and I were back and forth for a while. Never were we drafting each-other though. I always stayed next to him as I thought it safer and for sure more "fair." That was a fun ride. One of my favorites.