Same thing for me, gets bumped out of alignment regularly and the view is blocked by the pannier. Although, I usually only use the right side just for this reason. As far as hitting it with the knee, that may be due to my frame size and geometry (Trek Pilot, 52cm). I have found that I hit it starting off with my knee at the 9-10 o'clock crank position. Granted, I haven't had it on all that long either.
Overall, it is good for a 'heads up' and especially useful at nighttime. It's like driving, check mirror, shoulder check, change lanes. Mirror is just the first step, if it's the only step you take, good luck
FWIW
Kerry