Had to do that when I broke a spoke on a commute, but if they're all intact, I'm more likely to stop and true the wheel on the spot than unhook the brakes. Too many times I use the rear brake alone while signalling a left turn, and having the handle go slack would likely be an unpleasant surprise in a situation where I really don't need those.