This site uses cookies to improve your experience and to provide services and advertising. By continuing to browse, you agree to the use of cookies described in our Cookies Policy. You may change your settings at any time but this may impact on the functionality of the site. To learn more see our Cookies Policy.
OK
Dublin: 8 °C Monday 22 October, 2018
Advertisement

Facebook explains epic late-night outage

The 150-minute downtime was down to a process correcting an error, which threw up an error, which it tried to correct…

Image: @Twitterwhale via Twitpic

FACEBOOK HAS EXPLAINED the cause of the technical fault that caused it to go offline for about two-and-a-half hours last night – an outage it says was its longest downtime in four years – and surprisingly, it seems the outage was a little bit like the woman who swallowed a fly.

In a blog post written by engineer Robert Johnson, the network said the site’s downtime was as the result of a rogue system which tried to resolve a routine error in the site’s cache, but accidentally caused much more damage than it fixed.

The change in the cache had come about as a result of a manual change made by the site’s engineers, and when individual machines tried to correct the ‘incorrect’ data, the site’s databases were immediately overwhelmed.

Furthermore, each error was interpreted as an invalid response, which in turn the systems tried to correct – causing an exponentially large problem, with every error generating dozens more, akin to a speaker amplifying its own feedback noises.

The site says it’s turned off the rogue correction procedure and will work on an alternative mechanism.

The downtime last night triggered typically massive stampedes on Twitter; searches on the microblogging platform showed the word ‘Facebook’ featuring in almost 15,000 tweets a minute at about 10pm last night, shortly before the site went back online.

At the end of the downtime, topics relating to Facebook occupied most of the top spaces on Twitter’s trending topics list.

  • Share on Facebook
  • Email this article
  •  

About the author:

Gavan Reilly

Read next:

COMMENTS

This is YOUR comments community. Stay civil, stay constructive, stay on topic. Please familiarise yourself with our comments policy here before taking part.
write a comment

    Leave a commentcancel