see if cable is binding in the housing..

Internally there is just a cam that spreads the brake shoes out,
and return coil springs bringing them back together..

Automobile drum brakes in miniature.

IDK what levers you use.. try V brake levers for their greater cable pull length.

[I use old campag road levers , with my Mustache bars on the bike that has drum brake hubs]

(its my icy road studded tire old mountain bike.. )

UK made Sturmey archer Elite sealed bearing ..

to snappy up the return on a lever that did not have
it's own return spring, I put a coil spring around the cable between the cam turning lever
and the adjusting barrel/cable stop..

spring came from the hardware store ..