You could use a chain tensioner of some sort, like the Surly one. It bolts to the derailleur hanger and maintains chain tension using a spring mechanism like the one in a derailleur, but it hasn't got the shifting mechanism so it's lighter and simpler.