You should increase your calorise slightly on days you exercise. Your deficite on those days is too great and it's catching up to yo uon your rest days. Working out on such little calorise will lead to a serious burnout pretty soon, trust me, I did it to myself at the begining of the summer. I found that if I at a little more on workout days my muscles felt better and I still lost weight.