A data center outage sounds comfortably hypothetical, or like a brief and harmless setback, until it happens to you. In 2016, research from the Ponemon Institute found that data center downtime costs as much as $9,000 per minute. The signs seem to indicate that the costs will continue to rise — and no wonder, given the proliferation of new threats like data breaches and ransomware attacks.
It’s way past time to start thinking about unplanned data center downtime as a credible threat. Here are five ways to protect yourself from the inconvenience and the potentially sky-high price tag.
Take Stock of Your Immediate Surroundings
Let’s begin with one of the more unthinkable types of downtime: the kind that’s due to a natural disaster like a hurricane. Many regions in the world are prone to dangerous weather, and changes in the climate make extreme events like “named storms” even more likely.
As far as flooding best practices go, it’s wise to seek out a site for a new data center that’s at least 150 feet above sea level. And if you’re working with an existing building that’s acceptably above sea level but falls short in other areas, bring in structural engineers to make sure the building is resilient enough. For an example, even the glass panes across the front of Emerson Global Data Center in St. Louis can withstand a category F-3 tornado.
Prevent Common Sources of Human Error
By some accounting, human error is responsible for as many as 22% of all unplanned downtime events in data centers. That’s actually more than twice as common as weather-related downtime (10%) and almost four times as common as generator failure (6%). So what can we do about it?
Recall one story from May 2017, when a single British Airways data center engineer accidentally canceled more than 400 flights and brought 75,000 passengers’ journeys to a screeching halt. He accomplished this feat by unplugging one power supply. The mistake didn’t just disrupt business — it also severely damaged nearby IT equipment when it triggered a power surge.
The takeaway from that story is that training, mindfulness and accountability are paramount in company culture. The engineer in question was authorized to perform work in that part of the facility, but not to work with the power supplies. Overstepping his authority cost $112 million USD in lost equipment, productivity and business.
Find a Reliable Backup Power Source
Operating a data center means maintaining 100% uptime — or else watching your clients pack up their data and taking it to somebody who can. The best way to help make sure your data center operations continue unabated by inclement weather or disruptions to the power grid is to find a reliable source of backup power.
Remember that the cost for every minute of downtime is nearly $9,000. To make sure you don’t lose a single second, take some time to find a dependable backup generator plus a fuel provider with event response and priority agreements, no-run-out guarantees and other disaster preparedness services to help you feel confident about whatever the weather throws your way.
Protect Yourself Against Cybercrime
Let’s return once more to the Ponemon Institute’s downtime study. In 2010, cybercrime was to blame for just 2% of outages in data centers. In 2013, it rose to 18%. The figure stood at 22% in 2016, making this the fastest-growing cause of downtime among data center service providers.
Thankfully, the availability of automation tools, built with AI, can help take some of the workload off your engineers’ shoulders. In fact, Gartner predicts that, by 2020, a substantial number of data centers (as many as 30%) which choose not to deploy AI will fail to remain realistically competitive. But why?
Among other things, AI can learn regular traffic patterns and events across data centers and servers and then alert decision-makers about signs of possible intrusion or other foul play.
Artificial intelligence has emerged as a great tool in the pursuit of lower emissions and energy expenses, too. Google and other companies use AI to automatically fine-tune the climate control in their data facilities to make sure they’re not spending more than they need to — and also to make sure they don’t draw more power than is required, and risk an outage.
Make a Plan
For companies that operate data centers, nothing is more damaging to your public image than unplanned downtime. In this industry, even more so than others, just one lapse in availability or performance is enough to spell disaster.
That’s why the fifth action item here is probably the most important: make a plan. If you do face an unexpected outage, it falls on you to make sure everybody knows their role in getting services and infrastructure restored, contacting customers and clients, running analyses to find the cause and roll out a fix, and perform general damage control across the business.
This isn’t necessarily a pleasant topic, but it’s a vital one. With any luck, you feel a bit better prepared now to protect yourself against some of these most common sources of data center outages.
About Kayla Matthews
Kayla Matthews writes about data centers and big data for several industry publications, including The Data Center Journal, Data Center Frontier and insideBIGDATA. To read more posts from Kayla, you can follower her personal tech blog at ProductivityBytes.com.