Yes, you should be worried. If it's not already, your hub will soon be be gargabe. It may be that you installed them poorly, or a quirk of the cog / lockring combo you are using. What brands are you using? Sometimes they don't play well together.
Or maybe it's nothing to worry about. Who can tell.