You'll be fine. You're just going to be more likely to fall off (and it isn't nearly as bad as you think, trust me).

Started with the fixed-gear, it's a lot easier. It will carry you through the dead spots in your spin (which is the least stable part when you are on rollers).