Try your existing mirror above your glasses and turned outward horizontally, instead of your current position below the lens and vertical. That's where I have mine.

When I'm wearing an unzipped jacket, the wind puffs up the shoulders, and then my rear vision is blocked. That's probably similar to your problem. I have to sit up a bit to use the mirror.