It happens but I think it's pretty rare that someone would draft a commuter with panniers. Think of it as a compliment.