Every sport is tough. Olympic Weightlifting is tough because all of the hundreds of hours under the bar comes down to 6 attempts on the platform. Swimming is tough because those cats get up at 3 in the morning to drink bleach water and practices last forever. Triathlon is tough because you have 8 hours of time trialing...
Mass start bicycle racing does have an agony factor that is sport specific. You never know what the race will bring, it might be fast or slow, the pace might be erratic, terrain plays a role. Hold a wheel can be excruciating but then the pace might ease.