So while loosening the pivot bolt, the ball bearings unseated themselves and scattered themselves on the garage floor. Doh! I managed to retrieve 7 of them (the cage holds 12).

Not sure how common this is as I've only disassembled a handful of vintage brakes. This is the first time I've encountered brake calipers with bearings. Are they important? Maybe I can get away with using washers in its place...