I've had this happen to me many times on my daily commutes. I generally try to turn my commute into a pseudo-workout, so I'm usually cruising much faster than most other people on the road. The occasional ******* will grab my wheel, just as the OP described, and even try to slingshot off of me. This usually does not end well and I pass them again in a couple hundred feet.

My worst experience was when one guy (silently) grabbed my wheel after I passed him, and gave me a surprise sprint attack out of nowhere. I caught back up to him, only to have him grab my wheel again and give me ANOTHER sprint attack. This happened about three more times. The last time I caught back up to him, he looked at me and said he was doing intervals... At that moment I wish I had a pair of nunchucks to hit him square across the face.

Totally not cool.