Exactly how ultralight do you want it to be? In other words, what's your budget?

Frame is the most problematic bit as track frames are pretty burly, and ultralight road frames have vertical dropouts. If a lightish Al track/fixed frame is good enough for you, you should have no problem.
If you want something under 1300g, or even around/under 1000, I dunno.