Advertisement

We need your help now

Support from readers like you keeps The Journal open.

You are visiting us because we have something you value. Independent, unbiased news that tells the truth. Advertising revenue goes some way to support our mission, but this year it has not been enough.

If you've seen value in our reporting, please contribute what you can, so we can continue to produce accurate and meaningful journalism. For everyone who needs it.

@Twitterwhale via Twitpic
Facebook

Facebook explains epic late-night outage

The 150-minute downtime was down to a process correcting an error, which threw up an error, which it tried to correct…

FACEBOOK HAS EXPLAINED the cause of the technical fault that caused it to go offline for about two-and-a-half hours last night – an outage it says was its longest downtime in four years – and surprisingly, it seems the outage was a little bit like the woman who swallowed a fly.

In a blog post written by engineer Robert Johnson, the network said the site’s downtime was as the result of a rogue system which tried to resolve a routine error in the site’s cache, but accidentally caused much more damage than it fixed.

The change in the cache had come about as a result of a manual change made by the site’s engineers, and when individual machines tried to correct the ‘incorrect’ data, the site’s databases were immediately overwhelmed.

Furthermore, each error was interpreted as an invalid response, which in turn the systems tried to correct – causing an exponentially large problem, with every error generating dozens more, akin to a speaker amplifying its own feedback noises.

The site says it’s turned off the rogue correction procedure and will work on an alternative mechanism.

The downtime last night triggered typically massive stampedes on Twitter; searches on the microblogging platform showed the word ‘Facebook’ featuring in almost 15,000 tweets a minute at about 10pm last night, shortly before the site went back online.

At the end of the downtime, topics relating to Facebook occupied most of the top spaces on Twitter’s trending topics list.