For some, the point in a race IS crossing the finish line. Let's say I want to run the Disney Marathon next year. Well that is a race. But some skinny dude from Kenya or Zimbabwe or something wins every year. Some of the worlds best runners are invited to run there.
So since I can't possibly win - I should not go?
What if you had a chance to be in the TDF? you would not go cause you can't win?
What if I view a race as a personal challenge! What if I really do not care who wins?
Anyhow - to the op just keep working. You may never have a "fast" avg speed. You might not ever be a top cyclist. But does it matter?