The 80mm stem with bullhorns is about right. I run my bullhorns + aerobars with a stem that is 2cm shorter than the stem I use for drop bars. This is normal because bullhorns move the hands much further forward of the stem than drop bars do. If you installed drops, your stem would be around 100mm, which would be appropriate.
Get drop bars and a 100 and 110mm stem and see what that does for you. Right now, I think the frame is fine.
Keep in mind that you won't be able to take a beginner class or race anything but time-trials with bullhorn bars. So getting some drop bars is important. You don't need "track" drops unless you plan on doing sprint specific events. Most track riders use road bars. Even this guy:
(photo by Larry Hickmott)