What do you mean when you say you wobble? Your body wobbles, or your wheels don't track in a straight line?

Is it a problem regardless of cadence or power, or do you find the issue exacerbated when you're riding very easily, and less of an issue when you're putting power to the pedals?

I've found that the faster I go, the easier it is to maintain a rail-straight line (also the easier it is to sit up right with the hands off the bars, though that may only be up to a certain speed). I've also found that it's much easier to bounce around a bit on the bike with a higher cadence with little power output. Dropping the cadence, upping the power, or both, tends to cement the person to the seat a bit more.

Spinning out on rollers with no resistance makes it much more difficult to ride "smoothly" than it does with magnetic resistance or something of the like. I can do threshold workouts on rollers with resistance, but without it I'd be spinning really quickly and wobbling quite a bit more simply because there's not enough force to keep me in place. I suspect this is true of most people.