It might depend on how hard you ride. If you ride hard, the chain rubbing against the derailleur could cause a greater loss of energy than the banging of the crank against the derailleur. The friction from the chain rubbing is likely to increase as your effort increases. On the other hand, the crank bangs the derailleur when you're applying more power to the opposite crank, so the crank banging the derailleur is probably not banging it hard. And that force decreases as your pedaling effort increases.

So if you're tooling around, let the chain rub. If you're mashing hard, let the crank bang.