• news-banner

    Expert Insights

Resilience vs. Recovery - How the Facebook outage highlights important lessons

The recent Facebook outage disrupted all of its key global platforms, including Instagram and WhatsApp -   attracting significant attention, and six-hour communication vacuum. 

Following this, Facebook published a very interesting press release setting out in detail what happened, why, and most importantly, how they were learning from the incident. This sort of public communication is a fascinating insight into the processes behind recovering from a major outage, and a shift in Facebook’s handling of communication. Had there been a significant loss of data, leading to a personal data breach, or heavy involvement of insurers in an incident, it is unlikely that this type of transparency would occur from a multinational.

What lessons can businesses learn from Facebook’s response and what legal and business issues does it bring into question?

What happened?

According to Facebook’s press release, the technical disconnections in its network, showed that the incident broke the tools normally used to investigate and resolve network outages.  Repair and restoration of service therefore required physical presence of engineers at data centres, and required access to the router hardware, software and configurations which are designed to be difficult to modify, even with physical access.

Facebook specified that bringing the data centres back online had to be done carefully, to manage increasing loads as a full power up could have bought about further system failures.  One of the key quotes in the article in the final paragraph in the press release stated:

we have done extensive work hardening our systems to prevent unauthorized access and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity but an error of our own making.  I believe a trade-off like this is worth it – greatly increased day to day security versus a slower recovery from a hopefully rare event like this.

Is there a balance to strike in effective cyber security?

Cyber is part of modern warfare.  Similar thinking exists in the military sphere, including for example, the designing of tanks, which involves trade-off in the so-called “iron triangle” holy trinity of mobility, protection and firepower. 

Here, the design of tanks can vary very significantly, depending on their function and the context in which they are used, and according to the offensive or defensive capabilities required. In the case of cyber and infrastructure protection, Facebook has suggested that the trade-off of more resilience and cyber protection is worth it, even if this slows down recovery of the systems in the unlikely or, at least, reasonably uncommon circumstances of human error or force majeure. 

There are some important lessons from this analysis that may be relevant to contracts and services reliant on technology infrastructure.

How can businesses limit the impact of a platform or IT outage? Legal terms and conditions considerations

Large enterprises, including hosting and infrastructure providers such as Amazon Web Services (AWS) and Microsoft, commonly exclude or limit their liability for service unavailability and may cover losses exclusively by way of service credits.  For businesses, the careful evaluation of the remedies available for service downtime is vital. In particular, as they relate to establishing whether termination rights are required for a so-called “catastrophic” failure, by which we mean an outage of sufficient duration that it may affect the viability of the customer’s operations .Whereas smaller businesses, with little or no bargaining power over supplier terms and conditions, must balance whether it is worth suffering a temporary outage of this nature once in a while against the trade-off for greater availability and security the rest of the time.

For many businesses, there is a choice to run their own IT, or to rely on smaller IT service providers, who may offer more attractive commercial terms or liability caps in the event of an outage, against  larger providers who promise greater resilience and robustness, backed with best of breed information security controls and IT certifications. 

This is a judgement that each business will have to take on its own account, but realising that if an incident occurs, what the consequences could be for a slower than anticipated recovery is important for business continuity planning and operational resilience provision. A wider review of the risk profile of a business can also be balanced with appropriate insurance cover for loss of business or business interruption.

A careful review of Service Level Agreements (SLAs) may be worth considering in some cases. In an incident of this nature, the actual point at which services could technically be deemed to become available, thereby stopping the clock for the purposes of service resolution and service credit duration, may not be the point at which the service is actually fully operationally restored.

Operational Resilience

Many organisations are now reviewing operational business continuity in the light of operational resilience, which is now a mandatory consideration for many regulated businesses. 

In particular, operational resilience requires businesses to assess realistically what will happen when services fail, rather than assuming that services can never fail.  It is essential that the recovery point objectives (RPO) and recovery time objectives (RTO) are realistically managed and understood in the light of incidents of this nature, so that those RPOs and RTOs are not unrealistically short and could imperil the business.

Implications for remote maintenance and “dark” or edge data centres

Finally, there is increasing focus on trying to ensure that networks and data centres, in particular “edge” sites, can be supported and maintained remotely.

Clearly, the implications of diagnosis tools and virtual or remote means of access, or even entry door controls being disabled during an incident must be considered very carefully, as we have seen by this global outage incident

The new generation of “edge” processing will, of necessity, require buildings and networks to be supported remotely, and for fully “dark” data centres or microsites to be deployed, simply in order to ensure timely and cost-effective means of maintenance.

In this case, the equation between infrastructure resilience, access, physical and cyber security will have to be examined very carefully to ensure that the right balance of protection is balanced against ease of incident resolution in a similar way as the “iron triangle” applies to military hardware.

Our thinking

  • Doing business in the UAE & Israel

    William Reichert

    Events

  • Charles Russell Speechlys expands commercial offering with the appointment of Rebecca Steer

    Rebecca Steer

    News

  • The Times quotes Gareth Mills on the CMA’s preliminary approval of the Activision Blizzard-Microsoft deal

    Gareth Mills

    In the Press

  • City AM quotes Gareth Mills on the CMA’s new set of principles for regulating AI

    Gareth Mills

    In the Press

  • Silicon quotes Gareth Mills on the UK consumer lawsuit against Google

    Gareth Mills

    In the Press

  • Bloomberg and The Washington Post quote Richard Davies on multiclub ownership in world sports

    Richard Davies

    In the Press

  • Product compliance and Brexit - UK Government concedes to CE markings indefinite recognition

    Jamie Cartwright

    Quick Reads

  • UAE and the Grey List: Brief Update

    Karl Masi

    Insights

  • A Summer of Sport - Top 5 Legal Considerations

    Anna Sowerby

    Insights

  • Has the Orpéa plan impaired shareholder's consent? - Le plan de sauvegarde d'Orpéa n'a-t-il pas vicié le consentement des actionnaires historiques ?

    Dimitri-André Sonier

    Quick Reads

  • The Express quotes Gareth Mills on the CMA’s report on competition in the groceries sector

    Gareth Mills

    In the Press

  • Reuters quotes Gareth Mills on the CMA’s deadline extension of the Microsoft Activision Blizzard deal

    Gareth Mills

    In the Press

  • Les défaillances en France proches de leur niveau de 2019 - French insolvencies close to 2019 levels

    Dimitri-André Sonier

    Quick Reads

  • Law.com International quotes Simon Ridpath on the use of AI in the legal sector

    Simon Ridpath

    In the Press

  • Casino Group: An agreement with investors and debt holders is expected at the end of July

    Dimitri-André Sonier

    Quick Reads

  • Raconteur quotes Caroline Swain on misleading pricing practices

    Caroline Swain

    In the Press

  • DIAC Issues First Annual Report

    Georgia Fullarton

    Quick Reads

  • PE Hub quotes Richard Davies on private equity interest in football and sports

    Richard Davies

    In the Press

  • Payment Expert quotes Janine Regan on the record £1bn fine against Meta over EU data protection violations

    Janine Regan

    In the Press

  • The Guardian quotes Gareth Mills on Microsoft lodging an appeal against the CMA’s decision to block its Activision Blizzard deal

    Gareth Mills

    In the Press

  • The Times quotes Gareth Mills on the EU’s approval of the Microsoft-Activision deal

    Gareth Mills

    In the Press

  • One year on: "Influencer Culture: lights, camera, inaction" remains astonishingly accurate

    Caroline Swain

    Quick Reads

  • UKTN quotes Gareth Mills on the CMA's review of the UK AI market

    Gareth Mills

    In the Press

  • Saudi Center for Commercial Arbitration publishes new Arbitration Rules

    Peter Smith

    Quick Reads

  • The Financial Times quotes Nick White on risks to content creators promoting counterfeits

    Nick White

    In the Press

  • The Daily Telegraph quotes Nick White on AI and 'workplace displacement'

    Nick White

    In the Press

  • Charles Russell Speechlys achieves record success in The Legal 500 2023 EMEA directory

    Patrick Gearon FCIArb

    News

  • WhatsAppGate - Should businesses be reviewing their social media policies?

    Anna Rogers

    Quick Reads

  • Dubai announces its plan to streamline the enforcement of civil judgments and arbitral awards

    Peter Smith

    Quick Reads

  • Sign of the times - the British record football transfer which very nearly didn't happen

    Pei Li Kew

    Quick Reads

  • No love (island) lost for the #muffboss – #ad is great but don’t forget the other ad rules

    Caroline Swain

    Quick Reads

  • Is it really against the law to share your Netflix password?

    Quick Reads

  • Omnichannel innovation essential in the face of outlet decline

    Caroline Swain

    Quick Reads

  • Brand owners now required to police influencers

    Katie Bewick

    Quick Reads

  • Ten Years Since The 2012 Saudi Arbitration Law: Where Are We Now?

    Peter Smith

    Quick Reads

  • Strike a Pose - Usain Bolt files legendary victory celebration as a trademark

    Henry Cuthbert

    Quick Reads

  • ITV takes the plunge and “couples up” with Ebay to dress love island contestants in pre-loved clothing

    Natalie Batra

    Quick Reads

  • Constructing a Blue-print for Electronic Execution – New Guidance from the Industry Working Group on the Electronic Execution of Documents

    Quentin de la Bastide

    Quick Reads

  • To flex or not to flex: comparing traditional offices with flexible office space

    Emma Preece

    Quick Reads

  • PRA to further scrutinise cloud computing in 2022

    Mark Bailey

    Quick Reads

Back to top