my last road bike was way too twitchy to do it too... but its replacement is stable as can be, even at speeds.
I am sure it is part rider but that wouldn't explain why I couldn't do it on one bike and then getting on another was able to the first time i tried.