Disaster Planning Includes Preparing for Supplier Failure

When a car knocked down a power pole in Duncan BC, the point-of-sale terminals went out at the Costco store in Victoria, 60 km away. When the internet failed in our office, our carrier blamed it on power outages, though there were no outages near our office. This is the age of being let down by our suppliers. Why is this happening and how should you protect yourself?

By Ellen Koskinen-Dodgson

Ellen Koskinen-Dodgson is an IT and Telecommunications Management Consultant, electrical engineer, author, speaker, media resource and Expert Witness. She is the President and Managing Partner of TMC IT and Telecom Consulting Inc.

What Is Happening?

You’ve moved your apps to the cloud and something bad happens to your cloud provider. You chose them because they could run your software from any of their multiple sites. That sounds safe. Then they get hacked or they experienced an “oops” in a network software upgrade. There are many things that can go wrong – we’re discovering more all of the time, such as a network outage on your connection to your cloud provider. You thought you were well protected and then you have no service. You soon realize that you are only one of many, many customers and the best that you get is “sorry”.

You may have designed for power resiliency. Heathrow Airport did, as covered in our article last issue [Heathrow Blackout]. They paid the extra cost to bring in power feeds from two separate substations, entering the airport along separate physical paths so no backhoe could take out all of their power. What they didn’t know was that the power company high voltage network was temporarily reconfigured for maintenance to feed both substations from the same 400kV transformer – which was the one that eventually caught fire and killed all power to both substations. The low voltage distribution part of the power company was unaware of the Heathrow Airport expectation of diversity so never saw the problem coming. Heathrow assumed they had diversity.

Similarly, data network carriers regularly design their networks for lower costs and inadvertently introduce a single point of failure risk. The Costco carrier’s network, serving their POS terminals, was backhauled over fibre to Vancouver as an efficiency measure so the broken cable in Duncan knocked out service to the carrier’s customers in Victoria.

We sometimes design in resiliency by buying service from two separate carriers, sometimes paying for separate pathways into the building as an added measure of reliability. What can be unknown to us is that carrier A will contract with carrier B to connect your site to them, another efficiency measure, so you effectively have a single carrier.

As Jaguar Land Rover recently found [Cyber Attack Impacts Entire Economy], it wasn’t their fault when their IT supplier got hacked. That decision to outsource may have saved a lot of up-front IT costs – but it was responsible for shutting down the company for several weeks with a direct loss in sales exceeding $80 million per week. No IT savings can match that.

Why Does This Happen?

In the business world our suppliers focus on improving efficiency in order to reduce costs. There is very little investment in improving reliability or resiliency. This is true for voice, mobile, data networking, internet, offsite storage, regular cloud and specialty cloud such as building security systems and other application software.

Lessons Learned

The first lesson is to realize that your suppliers will sometimes fail you and you are powerless to stop it. If you contract everything to one supplier, you may well save money in the short term. However, you not only have no diversity built in, but a problem with that single supplier can bring down your entire organisation, costing you a lot more money in the long term. Even If you design for diversity, your diversity may be unavailable when you need it. Your organization’s Business Continuity Plan and IT Disaster Recovery Plan should be updated to specify what should be done if contracted services are unavailable. Have a ‘Plan B’ ready to go.

The second lesson is that you need to develop a relationship with your suppliers if you are designing for resilience. The right relationship will raise the likelihood that your service provider will not reconfigure your service, even temporarily, to more easily allow for maintenance, without discussing consequences with you. Your goal should be to get their commitment to disallow reconfiguring that will remove contracted resilience to you, the customer.

The third lesson is to reduce the impact of restarting after a loss of service and to improve the speed of restarting. This requires an analysis of the restart process timelines for all systems and the associated impacts to decide if more systems or components should be included in backup power.

If you’d like to discuss how TMC can help you independently assess the risk of failure that you face, or to comment on this article, please email me at ellen.