Does it make a difference? yes. It is signficant? depends on your definition of significant, and the size of your wallet. I used to TT on my road bike with clipons. I now have a dedicated TT bike, with more upright geometry (a Griffen Vulcan). It is definitely easier to get in an aero position on the Griffen, and I am faster with it. Also, there's the convenience factor of having the road bike set up for group rides and mass start events, and the TT bike set up for TT's. (Not insignificant on a day with a double stage in a weekend stage race).

If you're going to do one or two TTs or Tris, the setup you suggest can be quite serviceable.If your going to do it more often and can afford it, buy a TT bike.