Is Network Diversity an Illusion?

Years ago we expected that telecom (including data) networks were fully protected by redundant backups and diverse routing. Nowadays we see increasing numbers of major failures caused by single point events, like landslides. Why is this, and do we need to change how we build our networks as we place increasing reliance on them?

By Peter Aggus

Peter is a technology management consultant who specializes in security and radio systems. He has developed innovative & cost-effective solutions for clients in many industries.

What Is Going On?

A combination of a changing climate, with increased severe rainstorm events, and major forest fires are destabilizing slopes and causing increasing numbers of land and rock slides. Much of our telecom and power infrastructure in BC is overhead and is vulnerable to damage from these slides. Several years back, Vancouver Island was close to a total power outage that could potentially have lasted for many weeks because two high voltage transmission towers were in danger from major snow avalanches. During the atmospheric river event, the Vancouver area was in similar danger because the major Fraser Valley high voltage lines were in danger of flashover to the rising floodwater. Landslides and Road Traffic Crashes frequently wipe out wooden utility poles, causing extended power and communications outages.

These events are increasing – yet our power and telecom networks are designed to cost less rather than to be more resilient. In an age where we see competition as a good way of keeping prices down, we need to work out how to get the needed resilience. Must we, as end users, take on that responsibility, or can we regulate our utilities to do a better job?

Old School Network Design

Networks, both power and telecom are designed in two layers.

1. The local network connects end users to network hubs – sub-stations in power grids or Central Offices in telecom networks. This is considered the ‘Distribution Network’.
2. The network hubs are connected together by what can be considered as the ‘Primary Network’.

Distribution Networks are designed as a tree-and-branch structure to keep costs down. Much of the Distribution Network serves individual users, or small groups and it has always been difficult to justify major expense to increase resilience. Primary Networks, however, are shared between lots of users and they tend to be designed with alternate routes as backup, such that the loss of an individual link will not cripple service.

Modern Network Design

Power networks now include lots of small capacity generator systems, including solar and wind turbines. This distributes power sources more widely, increasing diversity of supply, but it makes stable network control a lot more difficult, as Spain recently found out. The source of their power grid instability has yet to be confirmed, but the consequence relied on lots more power sources to resynchronize, which resulted in total failure of the grid and a country-wide blackout.

Telecom networks are switching over to fibre in a big way, which has the unintended consequence of vastly increasing the size of the Distribution Network – largely because network is now much cheaper than switching. This leads to fewer much larger CO switching points, which reduces cost. As the main telecom network migrates to IP, these CO nodes are basically large routers. The Primary Network interconnecting them is, as always used to be the case, well protected. The down side is that much of the (now much larger) Distribution Network is one massive ‘single point failure’ risk. There may be diversity between network nodes but one cable fault will wipe out service to all downstream users. Recently, a crash on the Trans Canada highway south of Duncan on Vancouver Island removed service for all users connected to a network provider who routed all their Victoria area customers via that route over to a main node in Vancouver. With no backup, service restoration had to wait until the crash was cleared and crews could install new poles and fibre.

Future Options

These outages are set to continue unless steps are taken in mitigation. As we rely more on power and telecom, the impact of such outages will become more and more significant to many more users. Power and telecom is no longer an optional luxury.

One way is for federal regulators to force power and telecom providers to make their networks more resilient. The other way is for users to accept that resilience is not to be taken for granted but rather something that needs to be designed in by having feeds from multiple suppliers. Even that diversification is no guarantee because suppliers share infrastructure to varying degrees and their common mode failure risks are not even visible.

If you’d like to discuss network security, or to comment on this article, please email me at peter.