Resilience vs. Recovery - How the Facebook outage highlights important lessons
The recent Facebook outage disrupted all of its key global platforms, including Instagram and WhatsApp - attracting significant attention, and six-hour communication vacuum.
Following this, Facebook published a very interesting press release setting out in detail what happened, why, and most importantly, how they were learning from the incident. This sort of public communication is a fascinating insight into the processes behind recovering from a major outage, and a shift in Facebook’s handling of communication. Had there been a significant loss of data, leading to a personal data breach, or heavy involvement of insurers in an incident, it is unlikely that this type of transparency would occur from a multinational.
What lessons can businesses learn from Facebook’s response and what legal and business issues does it bring into question?
What happened?
According to Facebook’s press release, the technical disconnections in its network, showed that the incident broke the tools normally used to investigate and resolve network outages. Repair and restoration of service therefore required physical presence of engineers at data centres, and required access to the router hardware, software and configurations which are designed to be difficult to modify, even with physical access.
Facebook specified that bringing the data centres back online had to be done carefully, to manage increasing loads as a full power up could have bought about further system failures. One of the key quotes in the article in the final paragraph in the press release stated:
“we have done extensive work hardening our systems to prevent unauthorized access and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity but an error of our own making. I believe a trade-off like this is worth it – greatly increased day to day security versus a slower recovery from a hopefully rare event like this. “
Is there a balance to strike in effective cyber security?
Cyber is part of modern warfare. Similar thinking exists in the military sphere, including for example, the designing of tanks, which involves trade-off in the so-called “iron triangle” holy trinity of mobility, protection and firepower.
Here, the design of tanks can vary very significantly, depending on their function and the context in which they are used, and according to the offensive or defensive capabilities required. In the case of cyber and infrastructure protection, Facebook has suggested that the trade-off of more resilience and cyber protection is worth it, even if this slows down recovery of the systems in the unlikely or, at least, reasonably uncommon circumstances of human error or force majeure.
There are some important lessons from this analysis that may be relevant to contracts and services reliant on technology infrastructure.
How can businesses limit the impact of a platform or IT outage? Legal terms and conditions considerations
Large enterprises, including hosting and infrastructure providers such as Amazon Web Services (AWS) and Microsoft, commonly exclude or limit their liability for service unavailability and may cover losses exclusively by way of service credits. For businesses, the careful evaluation of the remedies available for service downtime is vital. In particular, as they relate to establishing whether termination rights are required for a so-called “catastrophic” failure, by which we mean an outage of sufficient duration that it may affect the viability of the customer’s operations .Whereas smaller businesses, with little or no bargaining power over supplier terms and conditions, must balance whether it is worth suffering a temporary outage of this nature once in a while against the trade-off for greater availability and security the rest of the time.
For many businesses, there is a choice to run their own IT, or to rely on smaller IT service providers, who may offer more attractive commercial terms or liability caps in the event of an outage, against larger providers who promise greater resilience and robustness, backed with best of breed information security controls and IT certifications.
This is a judgement that each business will have to take on its own account, but realising that if an incident occurs, what the consequences could be for a slower than anticipated recovery is important for business continuity planning and operational resilience provision. A wider review of the risk profile of a business can also be balanced with appropriate insurance cover for loss of business or business interruption.
A careful review of Service Level Agreements (SLAs) may be worth considering in some cases. In an incident of this nature, the actual point at which services could technically be deemed to become available, thereby stopping the clock for the purposes of service resolution and service credit duration, may not be the point at which the service is actually fully operationally restored.
Operational Resilience
Many organisations are now reviewing operational business continuity in the light of operational resilience, which is now a mandatory consideration for many regulated businesses.
In particular, operational resilience requires businesses to assess realistically what will happen when services fail, rather than assuming that services can never fail. It is essential that the recovery point objectives (RPO) and recovery time objectives (RTO) are realistically managed and understood in the light of incidents of this nature, so that those RPOs and RTOs are not unrealistically short and could imperil the business.
Implications for remote maintenance and “dark” or edge data centres
Finally, there is increasing focus on trying to ensure that networks and data centres, in particular “edge” sites, can be supported and maintained remotely.
Clearly, the implications of diagnosis tools and virtual or remote means of access, or even entry door controls being disabled during an incident must be considered very carefully, as we have seen by this global outage incident
The new generation of “edge” processing will, of necessity, require buildings and networks to be supported remotely, and for fully “dark” data centres or microsites to be deployed, simply in order to ensure timely and cost-effective means of maintenance.
In this case, the equation between infrastructure resilience, access, physical and cyber security will have to be examined very carefully to ensure that the right balance of protection is balanced against ease of incident resolution in a similar way as the “iron triangle” applies to military hardware.
Our thinking
Natalie Batra
ITV takes the plunge and “couples up” with Ebay to dress love island contestants in pre-loved clothing
Grégoire Uldry
New Swiss succession law on the transfer of businesses
On 10 June 2022, the Federal Council adopted its Message amending the Civil Code on the transfer of businesses by succession.
Louise Paterson
Artnet quotes Louise Paterson on the Ivory Act
UK’s Ivory Act comes into force
Pei Li Kew
Pei Li Kew writes for Pharmacy Business on the link between pharmacy and IP
Pei Li Kew writes for Pharmacy Business on the link between pharmacy and IP
Jonathan McDonald
Jonathan McDonald provides comment for City AM on the Data Reform Bill announced in the Queen's Speech
Jonathan McDonald provides comment for City AM on the Data Reform Bill announced in the Queen's Speech
Nick White
Charles Russell Speechlys advises Symphony Holdings Limited on the sale of its PONY trade mark portfolio for USD $28 million
Charles Russell Speechlys advises Symphony Holdings Limited on the sale of its PONY trade mark portfolio for USD $28 million.
Simon Ridpath
Simon Ridpath featured in the Lawyer’s Hot 100 list
Simon Ridpath features in The Lawyer’s Hot 100 list
Natalie Batra
Patents and Peppa Pig: What is happening to intellectual property rights in Russia?
Certain Russian individuals and businesses can now use patents, utility models and industrial designs without obtaining prior permission.
Simon Green
International Bar Association quotes Simon Green on the future of the legal sector in Hong Kong
International Bar Association quote Simon Green on the future of Hong Kong's legal sector
Charlotte Duly
Charlotte Duly quoted in Retail Gazette on House of Zana trademark dispute
Charlotte Duly quoted in Retail Gazette on House of Zana trademark dispute
Caroline Greenwell
Nowhere to hide for greenwashing brands
In the UK, regulators are cracking down, with many companies now at risk of financial and other penalties.
Jamie Cartwright
Weighing up the Plastic Packaging Tax
The Plastic Packaging Tax came into force on 1 April 2022.
Jamie Cartwright
Crunching numbers - Mandatory calorie laws come into force
The Calorie Labelling (Out of Home Sector) (England) Regulations 2021 (the Regulations) are now in force.
Jody MacDonald
Liverpool FC’s Hero Club and the current state of play with football NFTs
Liverpool’s Hero Club hit the headlines this week and serves as an interesting reflection of the current state of play.
Rachel Bell
Rachel Bell commented in IT Pro on the implications of the proposed EU’s Digital Markets Act
The proposed EU’s Digital Markets Act is set to require larger messaging platforms to interoperate with their smaller rivals.
Quentin de la Bastide
Constructing a Blue-print for Electronic Execution – New Guidance from the Industry Working Group on the Electronic Execution of Documents
Sonia Kenawy
Claimant ordered to pay security for costs in cryptocurrency dispute and digital assets rejected as form of security
Proceedings that are sure to be watched closely by the cryptocurrency community as well as legal practitioners.
Stewart Hey
Freezing Orders: Policing the Nuclear Option (PT 2)
Looking at the impact these checks and balances have when it comes to drafting and construing the terms of the order.
Stewart Hey
Freezing Orders: Policing the Nuclear Option
This article considered some of the checks and balances that apply when seeking access to one of the law’s most potent weapons.
Mark Hill
UAE Labour Law Update – The five big changes to note…
Federal Decree Law No. 33 of 2021 - The five big changes to note