That depends on how careful you're being in testing. If you're being really careful, you can reliably discern a difference in Crr *on the road* of maybe around 5% or maybe a bit less (I prefer to work in Crr rather that watts since watts depends on your speed and total weight). So if you're measuring a tire that has a Crr on the road of around .004, that means you can reliably discern a difference of around .0002, or a bit less.

So, one thing we noticed in addition to the rankings not changing much between roller and road testing: Real roads tend to be a little less perfect than even the kind of diamond plate Jarno uses for his big drum, plus when you pedal you're moving around and leaning the tire this way and that. So Crr measured in field tests tends to be maybe 1.5x higher in absolute terms than what Jarno measures on his drum. That is, the relative rankings and the relative differentials tend to be very close between rollers and road, but the absolute raw Crr values vary by maybe 1.5 (and this will depend on the condition of the pavement).

Here's the practical bottom line: if I absolutely positively need to know--like when I'm working with someone making a record attempt--I'll test CdA and Crr on the surface being raced on. However, for most purposes, I just use the roller tests, multiply by maybe 1.5, and use that. That usually gets me close (but, of course, I check).