A lot of it comes down to load. I ride a steel touring bike, which at 30 lbs is a good deal heavier than a true roadie, and has much less aggressive geometry. But riding that thing without a load, it's still a fast ride. Even though you can get a rack for a roadie, adding panniers does nasty things to your aerodynamics, and usually adds substantially to the load, all but negating the impact of the lighter frame. Additionally, low spoke count wheel and heavy panniers don't mix all that well, and you may run into trouble with such a setup.

To get the benefits of commuting on a roadie, you need to keep your luggage to a minimum, and carry it in such a way that it's not drastically changing your aerodynamics - a small saddle bag or a messenger bag probably. Plenty of people do it, and depending on where, how, and why you ride, it may work very well for you. Otherwise you'll just have an expensive bike that performs like a mid-range bike.