I checked inside and noticed that inner ball bearing retainer has cracked into many pieces. Cone and cup were already worn. Just took out metal chards and added more ball bearings, and extra graphite grease. That fixed problem.
Shimano should think about upgrading bearing systems as they are most vulnerable IGH components. Rohloff at least has enclosed bearings that can easily and cheaply replaced.