I do that. Sugino seat collar bolt, high end cable housings, small carbon parts (i.e steerer spacers), better handlebar tapes etc etc. and from a long time perspective I don't think it was ever worth it.

Ps. Still gonna be doing this because in a short time perspective the outcome of my bikes feels great!