Why take the aero bars off? The bike's geometry is optimised for using em, innit?
Seems to me, whatever reasons you opted for the drop bars, should apply to using a whole general-purpose road bike instead.
A TT bike isn't going to handle nearly as well, I'd have thought...