on part of the Internet's core infrastructure is the type of event that can galvanize the otherwise apathetic laity into outrage over the state of cyber security. We've all heard dire warnings about malefactors with various motivations taking out the electric grid, water treatment plants and the air traffic control system and it goes in one ear and out the other, but lose access to your Twitter feed or Spotify playlist and everyone becomes a security zealot. Last week's attack
Such disruptions are a wired-era example of how abstract problems aren't real until you're personally affected. As President Truman sagely observed about the economy, "It’s a recession when your neighbor loses his job; it’s a depression when you lose your own." However, someone else's misfortune can be a powerful motivator for positive change and enterprise IT can learn several valuable lessons from the recent spate of denial of service attacks.
The attack on DynDNS that disrupted access to many popular sites and an unknown number of businesses wasn't a surprise to security experts and definitely won't be the last. Indeed, if it were organized by something more than a group of vengeful hackers seeking to punish the company for its work exposing a shady DDoS mitigation firm, the attack could well be a trial run for something bigger. As I wrote a couple of weeks ago, there's little to stop future incidents that likewise exploit collectives of compromised IoT appliances.
...there is absolutely no incentive for a device manufacturer, or the user, to care about or invest in better security: hacked devices continued to work and owners were completely unaware of any nefarious activity.
I conclude that,
Cyber security is a global problem and the abuse of cheap, hackable IoT devices to create botnets is growing threat, however given the fact that international organizations have been unable to agree on the basics of cyber war and peace, I’m skeptical that effective multilateral regulations can be drafted in the near term.
The best that businesses can do is to prepare for the worst by protecting yourself both technologically and financially.
DynDNS and the power of distributed design
The target of last week's attack was attractive since DynDNS is a critical link for many businesses in connecting users to their online applications. Thus, taking out just one service provider can affect thousands of sites. However, DNS is not a weak link; it's a highly resilient, hierarchical protocol in which redundancy, service delegation and offline caching are built in. Much like IP networking itself, DNS is designed to work around failure. Thus, while the attack on DynDNS was undoubtedly more than enough to take out a Web site, it didn't disrupt access to DynDNS customers for everyone. As the company's initial explanation of the attack points out,
We should note that Dyn did not experience a system-wide outage at any time – for example, users accessing these sites on the West Coast would have been successful.
Even a second, more powerful wave and last-gasp follow-up failed to disrupt all operations,
This second wave was more global in nature (i.e. not limited to our East Coast POPs), but was mitigated in just over an hour; service was restored at approximately 1:00 pm ET. Again, at no time was there a network-wide outage, though some customers would have seen extended latency delays during that time.
News reports of a third attack wave were verified by Dyn based on our information. While there was a third attack attempted, we were able to successfully mitigate it without customer impact.
Although DynDNS has not detailed the mitigation steps it took, much of the credit must go to redundant, distributed design of both its infrastructure and the underlying DNS and network protocols that can thwart a concentrated attack on any single piece of infrastructure. A fascinating display of distributed infrastructure at work is the chart that one network engineer created that mapped changes in Internet routing tables as DynDNS attempted to direct traffic around servers that were under attack.
That's an important lesson for enterprise IT: critical infrastructure and applications must be designed and implemented using logically and physically distributed components that operate in a synchronicity that allows them to assume workloads of instances that fail or come under attack. It's a tall order full of architectural complexity and financial expense, but operating more like AWS or DynDNS itself will enable digital businesses to function in extreme situations, whether a natural disaster like Sandy or malicious attack like last week's.
Given the availability of utility-scale services, some or all of the distributed infrastructure should be hosted at one or more public cloud providers. Not only do they have far greater network, server and administrative capacity to handle incidents like the DynDNS attack, but they can provide mitigation and migration services that blocks or shunts malicious traffic and, assuming you have a redundantly-designed application, directs legitimate users to systems at other locales.
As I mentioned in discussing an earlier DDoS attack,
It took no less than the help of Google to restore Brian Krebs’ site by deflecting the stream of network noise that was choking his servers.
Financial preparation and my take
Like any unexpected calamity, malicious cyber attacks, whether DDoS or data theft, are a new class of business risk that organizations must assess and mitigate with insurance. Distributed, redundant infrastructure isn't cheap and amounts to an insurance investment, however as I detail in an earlier column, more traditional policies are available.
Cyber insurance is likely to become a routine cost of running a digital business and is something organizations need to start evaluating now, before disaster strikes.
As with other financial risks, the tradeoff between known, regular insurance premiums and unknown costs of cyber disruption and theft are unique to each organization; however, businesses most reliant on their online presence, digital data and intellectual property have the most to lose and will be the early adopters.
Although cloud services are an enabler of digital business infrastructure that can withstand cyber attack, the DynDNS incident highlights a significant risk of using shared services: your business can get caught in the crossfire when an attack targets your provider. The customers of DynDNS were collateral damage during last week's attack, although had any of them been individually targeted, it's unlikely they would have recovered as fast as DynDNS. Still, it underscores the importance of vetting service providers about their incident response plans and capabilities.
Although the mega cloud services like AWS, Azure and Google can withstand anything short of cyber world war, it's unclear how they would handle an attack targeting a particular customer. When do they choose to sacrifice the one for the good of the many and what is your recourse then? Cloud customers need to find out and plan accordingly.
IT organizations should use recent DDoS incidents to catalyze needed updates to disaster preparation, business continuity and information protection strategies that reflect the realities of today's digital economy and security landscape.