#Open journalism No news is bad news

Your contributions will help us continue to deliver the stories that are important to you

Support The Journal
Dublin: 2°C Saturday 28 November 2020
Advertisement

Facebook explains epic late-night outage

The 150-minute downtime was down to a process correcting an error, which threw up an error, which it tried to correct…

Image: @Twitterwhale via Twitpic

FACEBOOK HAS EXPLAINED the cause of the technical fault that caused it to go offline for about two-and-a-half hours last night – an outage it says was its longest downtime in four years – and surprisingly, it seems the outage was a little bit like the woman who swallowed a fly.

In a blog post written by engineer Robert Johnson, the network said the site’s downtime was as the result of a rogue system which tried to resolve a routine error in the site’s cache, but accidentally caused much more damage than it fixed.

The change in the cache had come about as a result of a manual change made by the site’s engineers, and when individual machines tried to correct the ‘incorrect’ data, the site’s databases were immediately overwhelmed.

Furthermore, each error was interpreted as an invalid response, which in turn the systems tried to correct – causing an exponentially large problem, with every error generating dozens more, akin to a speaker amplifying its own feedback noises.

The site says it’s turned off the rogue correction procedure and will work on an alternative mechanism.

The downtime last night triggered typically massive stampedes on Twitter; searches on the microblogging platform showed the word ‘Facebook’ featuring in almost 15,000 tweets a minute at about 10pm last night, shortly before the site went back online.

#Open journalism No news is bad news Support The Journal

Your contributions will help us continue to deliver the stories that are important to you

Support us now

At the end of the downtime, topics relating to Facebook occupied most of the top spaces on Twitter’s trending topics list.

About the author:

Gavan Reilly

Read next:

COMMENTS