LOL! I mean, uh, I hope you aren't hurt.
They definitely take a bit of practice to get going on and be able to stay on. A recommended strategy is to start in a hallway or doorway, so you can put your arms out to catch yourself if needed. And you must be smooth. Any jerkiness in your pedal stroke gets translated into side-to-side movement of the bike. Too far, and you're off, as you've noticed. The funny thing is that when you go off the rollers, the wheels are going however fast you were on the rollers - 20, 30, whatever, so they hit the carpet spinning, and you get the stench of burned fabric.
But after a few sessions, you'll find rollers as easy as regular riding, although you have to always pay attention. A benefit is that you'll be a better rider because of them. When you get good, try no hands.