I used to have this same problem at 12 and 24 hour solo mtb race. I tried playing around with fit and what not but the one thing that fixed the problem was switching to a light helmet. I ended up with a Giro Pro-Light, yeah it's expensive, but theres no plastic in the adjustment band to dig into your head and it weighs next to nothing.