Not always. If you find the puncture right away sometimes you can unseat the tire, pull out a small section of tube, apply the patch, and reseat without ever taking off the wheel. I'd be hard pressed to replace a tube more quickly than that. Maybe I'm slow at replacing a tube, but it definitely takes me longer than the patch method I just described (which admittedly can't always be used).

It's not a big deal if you prefer to replace the tube and not patch it. I'm surprised at how many people won't patch though.

One other reason I patch, is that if I'm riding with my wife and kids, I'd have to carry at least three sizes of tubes to cover everyone's wheel sizes -- and that's assuming I can get by with one that's not ideally sized if I need to.