They really aren't as difficult as you think. At least they weren't as difficult as I had imagined. I could ride normally after 15 minutes, though the first 5 minutes seemed impossible.
Because you have no forward momentum you can't crash into your TV at 20 mph. You would basically just fall over. And if you are like me, you did that plenty of times while learning clipless. It is no big deal. The fear is worse than the reality.
To prove this to yourself while off of the bike, just turn the cranks a few times in a high gear while just holding the bike off the ground. Then let the wheels touch the ground after it is going really fast. Does your bike fly off at 20 mph? No, the wheel just comes to a stop and the bike moves an inch maybe. Same deal with the rollers.
As with riding on the road you have to turn your front wheel to balance, but since the rollers create a smaller contact patch each movement is exagerated thus improving your handling skills and making the whole experience more interesting. I did fall off once last week, but somehow managed to end up with my rear tire still on the rollers and the front tire balanced on the 1/2" wide frame of the rollers. I didn't even fall over. After just 30 minutes of riding on the rollers I was able to watch TV without incident.