Resilience vs. Recovery - How the Facebook outage highlights important lessons

The recent Facebook outage disrupted all of its key global platforms, including Instagram and WhatsApp -   attracting significant attention, and six-hour communication vacuum. 

Following this, Facebook published a very interesting press release setting out in detail what happened, why, and most importantly, how they were learning from the incident. This sort of public communication is a fascinating insight into the processes behind recovering from a major outage, and a shift in Facebook’s handling of communication. Had there been a significant loss of data, leading to a personal data breach, or heavy involvement of insurers in an incident, it is unlikely that this type of transparency would occur from a multinational.

What lessons can businesses learn from Facebook’s response and what legal and business issues does it bring into question?

What happened?

According to Facebook’s press release, the technical disconnections in its network, showed that the incident broke the tools normally used to investigate and resolve network outages.  Repair and restoration of service therefore required physical presence of engineers at data centres, and required access to the router hardware, software and configurations which are designed to be difficult to modify, even with physical access.

Facebook specified that bringing the data centres back online had to be done carefully, to manage increasing loads as a full power up could have bought about further system failures.  One of the key quotes in the article in the final paragraph in the press release stated:

“we have done extensive work hardening our systems to prevent unauthorized access and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity but an error of our own making.  I believe a trade-off like this is worth it – greatly increased day to day security versus a slower recovery from a hopefully rare event like this. “

Is there a balance to strike in effective cyber security?

Cyber is part of modern warfare.  Similar thinking exists in the military sphere, including for example, the designing of tanks, which involves trade-off in the so-called “iron triangle” holy trinity of mobility, protection and firepower. 

Here, the design of tanks can vary very significantly, depending on their function and the context in which they are used, and according to the offensive or defensive capabilities required. In the case of cyber and infrastructure protection, Facebook has suggested that the trade-off of more resilience and cyber protection is worth it, even if this slows down recovery of the systems in the unlikely or, at least, reasonably uncommon circumstances of human error or force majeure. 

There are some important lessons from this analysis that may be relevant to contracts and services reliant on technology infrastructure.

How can businesses limit the impact of a platform or IT outage? Legal terms and conditions considerations

Large enterprises, including hosting and infrastructure providers such as Amazon Web Services (AWS) and Microsoft, commonly exclude or limit their liability for service unavailability and may cover losses exclusively by way of service credits.  For businesses, the careful evaluation of the remedies available for service downtime is vital. In particular, as they relate to establishing whether termination rights are required for a so-called “catastrophic” failure, by which we mean an outage of sufficient duration that it may affect the viability of the customer’s operations .Whereas smaller businesses, with little or no bargaining power over supplier terms and conditions, must balance whether it is worth suffering a temporary outage of this nature once in a while against the trade-off for greater availability and security the rest of the time.

For many businesses, there is a choice to run their own IT, or to rely on smaller IT service providers, who may offer more attractive commercial terms or liability caps in the event of an outage, against  larger providers who promise greater resilience and robustness, backed with best of breed information security controls and IT certifications. 

This is a judgement that each business will have to take on its own account, but realising that if an incident occurs, what the consequences could be for a slower than anticipated recovery is important for business continuity planning and operational resilience provision. A wider review of the risk profile of a business can also be balanced with appropriate insurance cover for loss of business or business interruption.

A careful review of Service Level Agreements (SLAs) may be worth considering in some cases. In an incident of this nature, the actual point at which services could technically be deemed to become available, thereby stopping the clock for the purposes of service resolution and service credit duration, may not be the point at which the service is actually fully operationally restored.

Operational Resilience

Many organisations are now reviewing operational business continuity in the light of operational resilience, which is now a mandatory consideration for many regulated businesses. 

In particular, operational resilience requires businesses to assess realistically what will happen when services fail, rather than assuming that services can never fail.  It is essential that the recovery point objectives (RPO) and recovery time objectives (RTO) are realistically managed and understood in the light of incidents of this nature, so that those RPOs and RTOs are not unrealistically short and could imperil the business.

Implications for remote maintenance and “dark” or edge data centres

Finally, there is increasing focus on trying to ensure that networks and data centres, in particular “edge” sites, can be supported and maintained remotely.

Clearly, the implications of diagnosis tools and virtual or remote means of access, or even entry door controls being disabled during an incident must be considered very carefully, as we have seen by this global outage incident

The new generation of “edge” processing will, of necessity, require buildings and networks to be supported remotely, and for fully “dark” data centres or microsites to be deployed, simply in order to ensure timely and cost-effective means of maintenance.

In this case, the equation between infrastructure resilience, access, physical and cyber security will have to be examined very carefully to ensure that the right balance of protection is balanced against ease of incident resolution in a similar way as the “iron triangle” applies to military hardware.