If your bike has the Specialized BG grips, try adjusting them.
I have mine tilted slightly up, but on my other bike, I had them set flat.
It may take a bit to find your 'sweet' spot.