I like to ride rollers. I have been riding for many years, have riddin rollers on and off through the years. I will admit that they seem harder to adapt to now then when I was 15!
I can't say whether or not there are people who just plain can't ride them, but it seems odd that you can't let go of the wall at all. Are you sure that you are absolutely upright when you attempt to ride. I have found that when I start out with my elbow against the wall, I am somewhat leaning toward the wall when I attempt to ride "wall free" and then I come very close to going down!
I will also say that for me there are days when I just don't feel comfortable on the rollers...usually feeling like my balance is a tick off...so I either cut it short or go on the trainer.
Bare