I'm not quite sure what sort of bikers you're talking about, but people may not want to race for a number of reasons. Referring more specifically to road cyclists, a person may be doing a base training ride, in which case the object is to keep his heart rate below a certain level (to build up the aerobic system). Racing during that stage might move him into the anaerobic heart rate range, which would sacrifice the quality of his training. Facing the tradeoff of sacrificing my training and either beating or being beaten by you versus sticking with my training plan to improve my riding in the long term, I'd probably go with the second option as well.

That said, I would suggest not interrupting a person just because he or she refuses to race. If I'm on a training ride, minding my own business, it would seem tremendously rude from my point of view if someone I didn't know just rode up, challenged me to a race, and when I refused attempted to hamper my training by slowing down in front of me.

Finally, if you don't think anyone can beat you, you should go grab a USCF license and race. If you're right, you'll surely be rolling in the dough soon enough.