It doesn't interfere at all. I only go that deep in the drop during descents and I need the brakes! My hand goes just under the clamp and I reach the brake lever with two fingers.

I put the mirror there because my other bike had bar-end shifters. I left it there because it gives a better angle of view than at the bar's end.