Hi Stapfam,
I was off-road a bit in the winter, even riding my mountain bike in the snow on some trails, and I think it did help me ride stronger once I hit the roads again. Also, I was riding at altitude (6,700 feet and above) so I think that worked well for me too.
Don't worry, the road bike strength will come back via the mountain bike!
Rick / OCRR