I can and do ride hands free but only when I need too, i.e. taking off a jacket or arm warmers, changing glasses, tearing open a Clif bar, etc. While I feel relatively secure with hands off, I do try to make it as infrequent as possible.
I haven't noticed any difference as I've gotten older (63 now).
Rick / OCRR