If they're wearing matching jerseys and riding their road bikes in a coordinated manner that would indicate that they were training, I would think that asking if they mind if I ride with them on my mountain bike would be a rather silly question. Of course they'll mind. It would be like trying to hop into the boat of a crew team.

That said, if he's just following them at a distance, there's no real reason to ask.