in this context 'bugger' is a more pithy version 'oh dear, what a pity', not to be confused with a rugger-bugger or buggery.