We started early, and when the "clear the way" vehicles came by we pulled over and watched the racers go by. After that we stay to the right as faster riders go by.
We will probably do the same next year. I don’t see a problem.
The only solution to your problem is to limit the number of riders.