I kind of have the same situation as you because I wear prescription glasses as well. For a while I bought sunglasses that fit over my glasses, but they are way too bulky looking and are uncomfortable.
What I have found to work the best for me is to have clip-on sunglasses lenses that fit directly on my glasses. That way I can more easily take them on and off.