Mark, this is my experience as well. There is a cost on a CX bike with no modification. BUT, you can change the characteristics of a CX bike to make it just as fast as a road bike though. Depends on the CX bike though. Some like the cross check are going to be heavy no matter what.

That's what great about CX bikes. Flip the stem, put 23's on and go play with your roadie buddies. Flip it back, put your cross tires on and go play in the dirt. Trade off is comfort.