I used to commute on a vintage Raleigh Team with 28C tires and a seatpost-mounted rack. Route was about half unpaved roads which often included washboards and soft spots, and one area which was "paved" with white rock with about 3" avg grain size. I started out using my MTB, but since the distance was around 12-15 miles, I really hated being so slow. The road bike worked out just fine and is still in use today.
However, I only did that for a year or two and didn't put thousands of miles on. Still, the road bike didn't suddenly fall over when it left the pavement or hit loose gravel, and didn't fall apart under the rough treatment. Ride quality was not much different than with the MTB (both unsuspended)--maybe because the road bike was lighter, and yes I knew when to stand up and I'm not afraid to "hop" either of them.
The only issue I had with the rack on the road bike was that the weight (CG) on the rack actually resided aft of the rear tire's contact point with the road, which made the front end a little light.