From what I read you are pleased with the bike when it is clean, but as it picks up grit and grime from use on dirt roads your satisfaction in the bike fails quickly. The simple solution is to stop riding on dirt, otherwise clean the bike after every ride to keep it working as your preference indicates.
FWIW, Apex has always sucked and is no comparison to any Di2 set up. This is coming from a guy that does not care for Di2, but has worked on plenty of Apex and Di2 bikes.