My wife and I use mirrors, our children (who are adults) do not.
If you use a mirror, you develop situation-assessment routines that become quite apparent when you try to ride without a mirror, which is to say, it's "uncomfortable".
For people like umd, my kids, and millions of other riders, it's just the opposite: they have non-mirror-using situation-assessment routines that work satisfactorily for them, and a mirror seems like an unnecessary encumbrance.