You may be taking breaths on your left side slower than on the right side. Pay close attention to your stroke rate during different breathing patterns and see if you're strokes on the left are slower.

If it's not a speed issue then perhaps you're a little less flexible in one of your shoulders.