The only problem I have with handbrakes is when an obstacle appears in front of me and I need to:

1) Slow down
2) Signal to change lane
3) Steer to avoid potholes and merge into next lane

...all in a short time interval.

Usually, I simply give up the signaling part.

I guess if you are on an obstacle course or riding on the sidewalk, then your hands would be preoccupied with steering.

But in traffic? I think braking and steering in traffic is not that hard, is it?