A cycling specific fleece cap and a neoprene ski mask that velcroes behind the neck work great for me. They hold the arms of my glasses snug against the sides of my face. I also have one a pair of these for extremely cold temps which I can wear over my glasses:
http://www.labsafety.com/search/goggles/24537158/
BL