Thanks island rider. On Saturday, I did an extended session on the rollers. I did one leg drills and high cadence drills and even some grinding on 53 12. I am beginning to conclude that I need to work on my core. It could be just that I waver too much on the saddle. On Sunday, I did a 50 km ride only focussing on keeping my line steady. Then I got on the rollers and put in all my concentration on keeping the bike straight, and I was able to ride one handed for about 4 rotations of the pedals.

I shall do more of these full concentration roller riding sessions and see.

By the way, my roller spindle is about 1.5 cm in front of the front wheel axle.