LiveJournal had a power outage. Looks like somebody hit the big red button in the data centre and everything stopped. It took a while to bring things back up again.

Then Brad Fitpatrick wrote a detailed explanation of what happened, and I find it interesting for two reasons:

  • Firstly as an example of business blogging where telling your customers what really happened and how you will fix it gives visibility and comfort. See some of the comments below the post.
  • Secondly as a lesson on how best attempts to make systems redundant often ends up not working. They had put some good energy into building a distributed system that would support failure of one part, but when the whole lot got turned off at once, it was difficult to get it all started again.

Link via BusinessLogs