Frame material and construction is also a big factor. Most road frame, save for some aero models, generally incorporate some desire to minimize road feedback to make for a more stable and comfortable ride. Actual track frames, say for example, a Cinelli Vigorelli, are designed for raw stiffness and power transfer at a minimal weight. However, when designing track frames, assuming they are actual track bikes and not road bikes with horizontal forkends, there is usually very little thought into making the bike more comfortable on rough pavement which you will invariably encounter on the road, since they expect that the bike will be ridden on the uniform surface of a velodrome. The result is a frame that if pushed into road riding will not only be uncomfortable on pavement but also structurally underbuilt for the stresses of road riding.

Long story short, thin walled track tubing = no good for less than perfect roads.