Generally you can work dirt down to the bottom bracket, where it tends to go anyway. Then you want to get it out via the seat tube. The trick to that is to rotate the frame through the horizontal axis, turning it flat, then upside down so the dirt drops into the seat tube instead of back up the down tube.

Generally, what you hear isn't dirt from the outside but bits of flux or braze that were left over from the build. Once the loose stuff is gone, it's over for the life of the frame.