Seriously, OP did not even pass them, so he lost the race.
Besides these two guys are probably like me when on a road bike:
See someone with panniers trying to race, I hold speed and let them catch up. When they are ahead of me, they always slow down by 2 mph and I draft them. I watch as they try to increase their speed and drop me, but to no avail. Once they are dead tired, I pull around them and sprint off. The next day, I check the cycling forums for the rant about the guy that drafted them without permission.