Sometimes i feel dizzy when i breath to my left side. I think it is because i swing my head faster on that side. Never a problem on my right.
Perhaps you are doing that as well - worth trying to move head slower.
Best of luck figuring it out.