I own 3 bikes:
a sporty cross hybrid with aerobar and cyclocross tires
a road/tri race bike with aerobar and 160TPI race tires
a TT bike with rear disc wheel, 404 front, 320TPI tubulars and full skin suit with TT helmet ... even shoe covers ;)
The difference is definitely there, but it's not as much as many people want to believe.
Riding at anaerobic threshold my typical fast cruising speeds are somewhere in the range of:
hybrid: 21 mph
road bike: 22 mph
TT bike: 23 mph
In a race ... that difference is like night and day.
In general everyday riding it is only a small difference especially considering that the wind direction's influence is many times greater.