if it doesn't bother you while you're riding then why care?
the only difference it'll make is in chain/cog life time, and if you only notice it when it's totally silent in your garage then i think the damage will be negligible...