They're complaining about what's making you mad anyway. You just happen to be the closest rider, so you catch the scorn. Brush it off. Just be smooth - sudden movements rightfully make people mad. Move up a couple spots every chance you get, it is so much easier near (but not at) the front. But then they'll yell at you for not working enough.