It has to do with the bike. My dept store Mountain bike is really steady hands free, my Cannondale EVO SuperSix is twitchy and took some getting used to. I can ride them both hands free for as long as I like but there is a huge difference.

I believe I get lazy and have poor form towards then end of my longer rides, as my lower back starts to ache. I ride hands free and sit up straight to stretch a little.