I would have slowed up. I'm not going to win by taking advantage of a rider fallen because of a spectator.
It would be different if it was something like... racing down a descent. If I went over the top of a mountain with a minute advantage over a rival, and I stormed down the mountain... if the guy behind me crashed, well then that is too bad. THAT is racing. His fault for making the mistake.
But if I wasn't racing to gain time on him right before it happened, then I wouldnt use the misfortune to gain time on him.