Well, I'll weigh in... I was a breaststroker in college - several time All-American blah blah blah... point is, I know breaststroke. (Gee, I sound so modest...)
As a beginner and not a very strong swimmer, doing breast occasionally throughout the swim is a fine idea. It will help you catch your breath, it will help you sight your target, it will help you feel more comfortable. It will also earn you some enemies if you're swimming in close proximity with others (you'll kick them). Many people do breaststroke or even backstroke sometimes, but I doubt there are many (any?) who do it for the entire swim. A more experienced triathlete could weigh in on this.
As a general rule, breaststroke is the slowest of the strokes. What I mean by that is that if you look at the fastest times (e.g. records) across the 4 strokes, breast is the slowest. That doesn't mean that it's YOUR slowest stroke. It might well be your fastest. However, I would doubt that you'd "be competitive with the crawl people" whilst doing breast.
Regarding being tiring... Breaststroke utilizes your legs much more than freestyle which demands more oxygen (big muscle groups). You reach anaerobic threshold doing breaststroke much faster than you do freestyle. That's why you see folks competing in 500, 1000, 1650 free, but the longest breaststroke event is a 200. Bottom line, when done properly, freestyle is much more efficient than the other strokes. Of course, this is all assuming that you're aiming for speed - if you coast and relax when you're swimming breast then the demands will be less on your body.
So... my advice to you is to continue working on your side-breathing technique and improve your freestyle. That is a very hard skill to learn, especially when you're an adult. See if you can get a lesson or two. But keep working at it and in the meantime, don't be afraid to fall back on breaststroke when you need it.
Hope this makes sense!