its a mix of material quality, construction quality(experience and skill of the frame builder), joining methods, and of course the suitabilty of the frame geometry for the type of riding.
the exact amount of importance to each of these is debatable, ie, an expert frame builder can make an excellent frame from gaspipe, how long it will last and how heavy it is, is not a reflection of the builder but of the material, since the builder is a master brazer and maker.
metalurgy is important, but its just one variable in a frame
you have to take the sum of all the variables to get the overall quality of the frame
but thats just my opinion