Maybe the outer dropouts aren't totally flat so in order to get a tight fit the pressure is not evenly distributed.
I added some lube on the pressure washed which is against the free-rotating housing - didn't help. I can get it to spin around if I loosen the lock but then it's not tight anymore.
EDIT: I was acutally able to get the housing to rotate with the help of using leather gloves for better grip
