• news-banner

    Expert Insights

Resilience vs. Recovery - How the Facebook outage highlights important lessons

The recent Facebook outage disrupted all of its key global platforms, including Instagram and WhatsApp -   attracting significant attention, and six-hour communication vacuum. 

Following this, Facebook published a very interesting press release setting out in detail what happened, why, and most importantly, how they were learning from the incident. This sort of public communication is a fascinating insight into the processes behind recovering from a major outage, and a shift in Facebook’s handling of communication. Had there been a significant loss of data, leading to a personal data breach, or heavy involvement of insurers in an incident, it is unlikely that this type of transparency would occur from a multinational.

What lessons can businesses learn from Facebook’s response and what legal and business issues does it bring into question?

What happened?

According to Facebook’s press release, the technical disconnections in its network, showed that the incident broke the tools normally used to investigate and resolve network outages.  Repair and restoration of service therefore required physical presence of engineers at data centres, and required access to the router hardware, software and configurations which are designed to be difficult to modify, even with physical access.

Facebook specified that bringing the data centres back online had to be done carefully, to manage increasing loads as a full power up could have bought about further system failures.  One of the key quotes in the article in the final paragraph in the press release stated:

we have done extensive work hardening our systems to prevent unauthorized access and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity but an error of our own making.  I believe a trade-off like this is worth it – greatly increased day to day security versus a slower recovery from a hopefully rare event like this.

Is there a balance to strike in effective cyber security?

Cyber is part of modern warfare.  Similar thinking exists in the military sphere, including for example, the designing of tanks, which involves trade-off in the so-called “iron triangle” holy trinity of mobility, protection and firepower. 

Here, the design of tanks can vary very significantly, depending on their function and the context in which they are used, and according to the offensive or defensive capabilities required. In the case of cyber and infrastructure protection, Facebook has suggested that the trade-off of more resilience and cyber protection is worth it, even if this slows down recovery of the systems in the unlikely or, at least, reasonably uncommon circumstances of human error or force majeure. 

There are some important lessons from this analysis that may be relevant to contracts and services reliant on technology infrastructure.

How can businesses limit the impact of a platform or IT outage? Legal terms and conditions considerations

Large enterprises, including hosting and infrastructure providers such as Amazon Web Services (AWS) and Microsoft, commonly exclude or limit their liability for service unavailability and may cover losses exclusively by way of service credits.  For businesses, the careful evaluation of the remedies available for service downtime is vital. In particular, as they relate to establishing whether termination rights are required for a so-called “catastrophic” failure, by which we mean an outage of sufficient duration that it may affect the viability of the customer’s operations .Whereas smaller businesses, with little or no bargaining power over supplier terms and conditions, must balance whether it is worth suffering a temporary outage of this nature once in a while against the trade-off for greater availability and security the rest of the time.

For many businesses, there is a choice to run their own IT, or to rely on smaller IT service providers, who may offer more attractive commercial terms or liability caps in the event of an outage, against  larger providers who promise greater resilience and robustness, backed with best of breed information security controls and IT certifications. 

This is a judgement that each business will have to take on its own account, but realising that if an incident occurs, what the consequences could be for a slower than anticipated recovery is important for business continuity planning and operational resilience provision. A wider review of the risk profile of a business can also be balanced with appropriate insurance cover for loss of business or business interruption.

A careful review of Service Level Agreements (SLAs) may be worth considering in some cases. In an incident of this nature, the actual point at which services could technically be deemed to become available, thereby stopping the clock for the purposes of service resolution and service credit duration, may not be the point at which the service is actually fully operationally restored.

Operational Resilience

Many organisations are now reviewing operational business continuity in the light of operational resilience, which is now a mandatory consideration for many regulated businesses. 

In particular, operational resilience requires businesses to assess realistically what will happen when services fail, rather than assuming that services can never fail.  It is essential that the recovery point objectives (RPO) and recovery time objectives (RTO) are realistically managed and understood in the light of incidents of this nature, so that those RPOs and RTOs are not unrealistically short and could imperil the business.

Implications for remote maintenance and “dark” or edge data centres

Finally, there is increasing focus on trying to ensure that networks and data centres, in particular “edge” sites, can be supported and maintained remotely.

Clearly, the implications of diagnosis tools and virtual or remote means of access, or even entry door controls being disabled during an incident must be considered very carefully, as we have seen by this global outage incident

The new generation of “edge” processing will, of necessity, require buildings and networks to be supported remotely, and for fully “dark” data centres or microsites to be deployed, simply in order to ensure timely and cost-effective means of maintenance.

In this case, the equation between infrastructure resilience, access, physical and cyber security will have to be examined very carefully to ensure that the right balance of protection is balanced against ease of incident resolution in a similar way as the “iron triangle” applies to military hardware.

Our thinking

  • Unlocking Opportunities: Introduction of the Re-domiciliation Regime in Hong Kong

    Shirley Fu

    Insights

  • UK Cybersecurity and Resilience Policy Statement April 2025 - Impacts for Managed Services Providers and Data Centres

    Mark Bailey

    Insights

  • Mastering Claims Against Estates: A Guide to Debt Enforcement in Switzerland

    Remo Wagner

    Quick Reads

  • The Economic Times interviews Kim Lalli on the UK-India Free Trade Agreement

    Kim Lalli

    In the Press

  • The new UK-India Free Trade Agreement – a significant development for both nations

    Kim Lalli

    Quick Reads

  • Token2049 week - what's on the horizon?

    Racheal Muldoon

    Quick Reads

  • Rebecca Steer writes for Artificial Lawyer on GenAI, copyright and the future of innovation

    Rebecca Steer

    In the Press

  • Computing quotes Gareth Mills on a major antitrust case involving Google

    Gareth Mills

    In the Press

  • Charles Russell Speechlys advises long standing client Puma Growth Partners on its investment in LOVE CORN

    Ashwin Pillay

    News

  • Global Insight quotes Shirley Fu, Tom Wong and Victoria Younghusband on trends in corporate activity in China

    Shirley Fu

    In the Press

  • PRC amends its AML Law to regulate specific non-financial institutions

    Shirley Fu

    Insights

  • New Government guidance for businesses on section 54 statements under the Modern Slavery Act

    Kerry Stares

    Insights

  • Buyouts Insider quotes Darren Bailey on private equity's increasing interest in sports

    Darren Bailey

    In the Press

  • Government received 11,500 responses to AI and Copyright Consultation

    Rebecca Steer

    Quick Reads

  • City AM quotes Darren Bailey on the compliance of NBA Europe's format and salary cap with EU law

    Darren Bailey

    In the Press

  • Dubai free zone companies can now access mainland

    Mo Nawash

    Quick Reads

  • Data protection in the UK: Charities anticipate green light to rely on direct marketing exemption

    Courtney Benard

    Quick Reads

  • UK Government’s Consultation on Copyright and AI: What’s Next for AI Developers and Creators?

    Rebecca Steer

    Insights

  • Charles Russell Speechlys ‘Client Conversations’ features Giles Pocock – VP of Brand and Marketing at Bowers & Wilkins

    Simon Ridpath

    News

  • Corporate Transparency Act: I’m still alive but I’m barely breathin’

    Timmoney Ng

    Quick Reads

Back to top