A lot of time it's the interface between crank and bottom bracket in my experience. Often one side will be just worn enough that they won't mesh just right and they'll give a little bit of play on one side or another. Normally it'll only be present when you're really cranking (top rings) and putting enough pressure on the cranks to cause it to shift.
It's exactly what happened to me with specialized crankset. So I replaced the crankset (but not the bottom bracket DOH) and had the same problem crop up relatively quickly. Local shop pointed out the wear on the bottom bracket and installed a new bb and found me a new unabused left crank. I just finally managed to kill that crankset but it was definately a lesson for me. I'm pretty religious about checking the tapers/splines/etc for odd wear patterns these days when I'm swapping cranks or replacing a bb.