Advertise on

User Tag List

Results 1 to 2 of 2
  1. #1
    Senior Member
    Join Date
    Aug 2006
    0 Post(s)
    0 Thread(s)

    Site outage horror story

    I'm glad that BF withstood the attacks, even though it took a bit to get back functioning. I'm glad it didn't suffer the horror story of another place that I used to frequent.

    A small, fairly esoteric forum site which ran on a LAMP (Linux, Apache, MySQL, PHP) stack. The admins knew what they were doing, and the site was backed up to tape nightly. I didn't have admin access to the box there (nor did I want them, because with the root password comes great responsibility), but I did have moderator status on their boards, and helped with the anti-spam methods.

    On a Friday night, the admins found out the machine was flattened. Apparently someone managed to get root and run dd if=/dev/zero of=/dev/whatever and zero out all the hard disks.

    It gets worse. The machine wasn't just rooted at that time where everything was taken down, but weeks to months earlier. For an unknown time beforehand, someone replaced the cronjob that did the backups (via tar) with a script that did a "dd if=/dev/urandom of=/dev/tape" so the machine looked like it was backing up, but wasn't. So, the admins swapped out tapes every so often in the usual cycle of backups, thinking that the backups were doing fine.

    So, pretty much all was lost from the machine, except someone who had an rsync of the bare box's config files (and thus ended up finding the changed cron job after the fact).

    Moral of the story: Verify your backups on another machine, take a tape out of the rotation at least quarterly, if not monthly to put on a shelf indefinitely. Offline backups that can be set to read only (either tapes, or burning to DVD) are very important.

  2. #2
    Wookie Fred chewybrian's Avatar
    Join Date
    Mar 2008
    East Florida
    My Bikes
    '66 Raleigh Superbe, 80 Nishiki Maxima, 07 Gary Fisher Utopia, 09 Surly LHT
    1 Post(s)
    0 Thread(s)
    Let me see if I get your point:

    The duct tape clogged the toilet and made it back up. डक्ट टेप टॉयलेट भरा है और इसे वापस कर दिया. और लीनुस और लुसी चार्ली ब्राउन के साथ फुटबॉल लात मार रहे थे. Linus and Lucy were kicking the football with Charlie Brown. शुक्रवार की रात, हम टर्मिनेटर देखा, और मशीन जीता. Friday night, we played Terminator on the XBox, and the machine won. टर्मिनेटर देखा, और मशीन जीता. सुनिश्चित क. Make sure you have backups for your offensive line, in case they get injured. के मामले में वे घायल हो गए हैं.

    Did I get it right?
    Campione Del Mondo Immaginario

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts