No matter how fast you're pedaling on rollers you have no speed relative to your surroundings, so the worst that'll happen is you'll fall over, which could be bad if you hit your head on something. When I'm riding rollers I set them up in a door way so that I could brace my self against the doorway to get started and if I ride off the rollers, I can catch myself on the doorway. They definitely will smooth out your steering.

Here's another tip. Practice riding no hands.