Sounds like your fitness level is high -- better than most people just because you ride at all, period -- but I wonder if core exercises (like planks or side planks and all that) would help. It might make balance adjustments happen sooner.