Double check your set up and make sure the front hub center line trails the front roller by a bit. Once you get the hang of it rollers make for a great workout.

The sheet rock repair can come later....

The house I lived in when I first had rollers was old and had plaster and lath walls...they don't break, collar bones do.

Aaron