I never had this problem before, so I'm really puzzled: I have a Shimano rear hub with their rollerbrake attachment (I think it's the Nexus standard, or Nexave, don't recall correctly). It's the exact same attachment as on Shimano's geared hubs, eccept this isn't geared, it's for a cassette.
So I installed the rollerbrake, but the rotating alluminum disk wobbles. I can easily push it in, but it won't stay there; after several revolutions of the wheel it goes out (towards the wheel) and wobbles. This creates a rhythmic sound that's not too disturbing, eccept I have no clue why this is happening. Braking is not affected in the slightest.