I ride my tri bike the large majority of the time. I ride my road bike for group rides (and I don't do many of those). I ride my fixed gear to the store or for short, easy rides. I suspect that your tri bike will get more comfortable on longer rides the more long rides you do on it. I can easily ride 100+ miles on my tri bike without being uncomfortable.
__________________
Fredo, you're my older brother and I love you...but don't ever take sides, with anyone, against the family again...ever.