Do quick releases wear out?

I have an older forged dropout bike with Sansin or Sunshine all metal QRs. I am running a modest 42x16t but from a stand still and going hard i still seem to pull my wheel sideways. I am wondering maybe if hte cam in my qr is worn or something. I guess i might need to try another one.

I have the thing horsed down really really good...meaning it takes almost everything i have to close it yet it still seems to allow that movement.