Is Your Data Center Prepared for a Natural Disaster?
By: Ross Warrington on April 23, 2020
The threat of a natural disaster looms large in the minds of data center providers around the world. Managing the power, cooling capacity, and security demands of a data center is difficult enough—even before taking threats such as hurricanes, earthquakes, or flooding into consideration. Unfortunately, a natural disaster isn’t something companies can afford to simply ignore, because maintaining data access in the event of a crisis can mean the difference between a company’s success and its failure.
While precise figures are difficult to come by, the Federal Emergency Management Agency (FEMA) estimates that as many as 40 to 60 percent of small businesses never reopen after a disaster. One of the key reasons for that is the loss of critical records and other data, but it’s also the case that most companies simply cannot afford to go “offline” for a period of time. If they’re unable to collect revenue, capitalize on new opportunities, and strengthen their brand, they will quickly lose out to competitors who have been more active about risk mitigation.
What the COVID-19 Crisis Reveals About Risk Management
Nothing has revealed the dynamic (and frightening) nature of organizational risk more than the COVID-19 pandemic. When the initial wave of outbreaks forced many states to issue social distancing guidelines and then stay at home orders, organizations across a range of industries were abruptly forced into remote working arrangements that their IT infrastructure and cybersecurity controls simply weren’t prepared to handle. Stores across the country were forced to shut down, leaving any companies that had neglected their online sales presence in recent years at a pronounced disadvantage.
The problem was even worse for manufacturers and healthcare organizations, which were hit by a one-two punch of supply chain disruption and increased demand for services. Companies that relied upon overseas shipments found their distribution centers empty, even as online customers were demanding essential supplies. With limited flexibility, they found themselves unable to meet that demand. Healthcare providers were already struggling to keep up with the rapidly increasing number of COVID-19 patients, but they also felt the impact on their digital infrastructure. With more people making use of telemedicine and medical professionals using healthcare cloud infrastructure to share research data, having secure and reliable IT systems with high levels of uptime became even more crucial than normal.
From a data center standpoint, the drastic increase in internet usage across the country put severe pressure on network infrastructure. Verizon alone saw internet usage spike 20 percent in the span of a single week, and the combined number of worldwide service outages over a six week period increased by more than 40 percent. Those combined figures expose any organization without a reliable data center solution to substantial downtime risk at the precise moment when they need their network services more than ever.
Beyond COVID-19: The Possible Effects of a National Disaster on Your Data Center
It’s imperative, then, that every data center has a comprehensive plan for mitigating risk and protecting data in the event of a natural disaster. While every effort should be made to maintain server uptime, data centers must consider the possibility that they won’t be able to deliver on that promise in a disaster situation. After all, even the most robust SLA won’t amount to much if the power fails (along with backup systems) and no one can physically access the data center to get critical systems running again.
Here are just a few examples of natural disasters that left data centers reeling:
It’s worth reemphasizing just how disruptive the COVID-19 outbreak has been for many data centers. Most business continuity and disaster recovery plans assume that natural disasters are specific, localized events that will have, at worst, impacts on the regional level. What the coronavirus pandemic has demonstrated, however, is that some risks have the potential to be truly global. Data centers around the world have been forced to adjust their operational procedures on the fly to maintain consistent operational uptime while also meeting the demands of social distancing requirements. Providers that wisely made investments in remote monitoring technology (like vXchnge’s in\site platform) and automated systems have been better positioned to deal with the crisis than others.
They say lightning doesn’t strike the same place twice, but in 2015 one of Google’s European data centers was struck by lightning not once, but four times, causing errors in 5% of the disks responsible for Google Compute Engine (GCE) instances. Although the company restored many of the drives, an estimated 0.000001% of data stored in the data center was irrecoverably lost. While that might not sound like much, try telling that to the customers who were affected by it.
According to National Geographic, 2017 was the most expensive hurricane season in U.S. history, costing roughly $200 billion. With their combination of high winds, storm surge, and heavy rains, hurricanes are one of the most dangerous natural disasters data centers must contend with. The sudden flooding resulting from Hurricane Sandy in 2012 caused extensive data center outages in New York and New Jersey. These failures were made even worse by the fact that backup systems were located in the same geographic region and were knocked out by the same weather event. With climate change contributing to more intense weather events, organizations can no longer afford to ignore hurricanes when assessing potential risks.
A devastating 2011 tornado ripped through several hospital buildings in Joplin, Missouri, one of which was a data center. While none of the data lost was mission-critical, that was only because most of the information stored there had been migrated to a new offsite data center just a few weeks earlier. Hospital officials noted that if the tornado had hit a month earlier, the data loss would have been catastrophic and rendered the hospital completely inoperable. Given the frequency of tornadoes in much of the country, it’s important for data centers to take this risk into consideration.
Severe flooding in Leeds, UK caused a Vodafone data center to temporarily lose power during Christmas of 2015. While data loss was negligible, the power outage disrupted mobile phone service temporarily. Vodafone, of course, has a bit of history with flooding, having suffered one of the most infamous data center disasters when its Istanbul data center was devastated by flooding in 2009.
So far, data centers have been lucky. Modern architectural standards and additional precautions (such as special enclosures and rollers for server racks) have gone a long way towards protecting data centers from earthquakes, even in high-risk areas.
Disaster planning is all about expecting the unexpected. Take, for instance, the squirrel that knocked Yahoo’s Santa Clara data center offline for several hours in 2010, or the truck that drove into a transformer feeding power into a Backspace data center in 2007.
Is Your Data Center Prepared?
The very first step any data center should take to prepare for disaster is to perform a comprehensive risk assessment. This review will identify both the likelihood and expected consequences of potential disasters. Once these risks have been identified, the provider needs to create a step-by-step checklist detailing what actions need to be taken in the event of each specific disaster. All relevant staff and personnel should familiarize themselves with these plans and conduct drills on a regular basis to ensure everyone knows what to do in a disaster scenario.
Keeping the network up and running is a key consideration in addition to preserving customer data. Every moment of downtime carries with it real financial costs. Any good data center should already have extensive network redundancies incorporated into its computing infrastructure, but these backup plans need to be even more reliable in a disaster situation.
Regular testing is essential to ensure that when power is disrupted and systems fail, customer data and critical operations are kept online and secure. Also, all relevant staff must be trained on what to do if the redundancy system doesn’t operate as designed. This normally involves critical manual intervention, and personnel must be trained in a ‘hands-on’ environment to gain the experience and understanding of the process and system reaction to the manual transfers.
Data centers also need to consider how disasters will affect the infrastructure around them. When Hurricane Sandy struck New York City, for example, many backup generators failed because they ran out of fuel and could not be replenished due to the flooded city streets. Because physical access to the data center could be limited and remote access may not be possible, it’s vital to have automated backup systems in place to ensure that customers’ mission-critical data and services are not lost or disrupted in the event of power failure.
If data is going to be backed up at another data center, that facility should be located far from the one being affected by a natural disaster. Data center providers with facilities spread across a wide geographic area are better able to ensure that a large scale natural disaster will not be able to take down all of their services. Of course, as the COVID-19 crisis has demonstrated, they should not fall into the trap of thinking that geographic distribution will protect their operations from every form of disaster.
As natural disasters like hurricanes and wildfires become more frequent, data centers must take active measures to protect their facilities and, by extension, their customers. Although cloud services make it easier to back up essential assets and modern construction techniques can better protect equipment from physical danger, data centers still rely upon factors outside their control to keep up and running. Only by preparing to deal with a disruption to their everyday operations can they truly be ready for whatever disaster nature decides to throw at them.
Prepare for Anything with vXchnge Risk Mitigation Services
Getting through the challenges of a natural disaster or unexpected crisis is easier when you have a trusted partner by your side. At vXchnge, we’ve designed every one of our data center locations with risk mitigation in mind to ensure that our customers can keep their services up and running no matter what’s happening beyond the walls of the data floor. With 100% uptime SLAs and the unprecedented visibility and control provided by the award-winning in\site intelligent monitoring platform, vXchnge colocation data centers can help you protect your brand when it matters most.
Ross is a Regional Vice President, Operations at vXchnge and is responsible for managing all 14 data center locations. With more than 30 years of experience, Ross has managed data center construction, engineering, repair and maintenance, leading him to the emerging business of colocation.
Subscribe to vXchnge Blog
Choose a Data Center Provider that Solves Problems and Adds Value.