I rarely think of challenging anyone. But I have to admit when an 18 year old kid on mountain bike taunts me, (a guy in his 50's), and challenges me a few times,and I'm on my fastest bike, it's almost impossible to resist. He asked for it.
You have to admit it's a nice feeling to drop or pass someone who really,really, asks for it.