yea it's definitely the stem on the adapter that's slipping.
i'm all for a little give but this thing slides as if it's been greased. like, i'll start pulling to go up a hill and by the time i'm at the top the bars are twisted a few inches to the side.