People comparing custom bicycles to clothing aren't taking their comparison far enough. Sure I can walk into a Target or Talbots and find something I can wear off the rack. Does it fit? After a fashion. Can I have it altered to improve the fit? Yes, within narrow limitations.
Compare that with a garment made to your measurements. You go to the local dressmaker or tailor, point to a pattern you like, select the fabric, and they make it for you. If they are good at their trade, they will make the necessary alterations to the pattern so the waist hits at the right spot, the shoulders are precisely the correct width. The finished garment feels more comfortable and looks better on you than anything you could buy off the rack.
Now, compare that to a bespoke garment. You start with a conversation and the maker learns what you are looking for. Using a wide knowledge base and wealth of experience, he or she discusses the limitations and advantages of different materials and styles to meet the parameters of your needs, bringing up aspects you hadn't considered, offering options you weren't aware existed, and designs an esthetic and functional garment for YOU. It might look like a well-cut and fitted business suit, but it fits so well that it feels more comfortable than your coziest loungewear. It FITS you, both physically and esthetically.
Is it worth it? Yes--for me it was. But it depends how you define "worth". Are you going to analyze the expense on a cost-per-ride basis or quantify the pleasure of riding the bicycle against what else you would have spent the money on?
Riding a standard production bike--even a well-fitted one--always feels "off" to me. Riding my hand-built bike is like pulling on a well-worn leather glove. It is so easy and comfortable that you focus on the riding, whereas with the production bike I am constantly aware of the niggling little areas where it just isn't comfortable.
The pitfall to avoid with a hand-built bicycle is expecting it to be perfect, and to do things beyond its design. After spending "that much money", you may irrationally think your sturdy commute bike should also be a nimble hill climber and a rough-and-ready mountain bike. A good builder will discuss the tradeoffs with you and the implications of your choices.
The other pitfall is once you learn how comfortable a hand-built bicycle can be, the production bicycles lose their appeal. You stop thinking that people with multiple hand-built bicycles are lunatics, and start thinking,"I'm nowhere close to what I spent to buy a car" and you start saving up for a second or third.