Solarwinds incident demonstrates 'weakest link' principle of IT security

Profile picture for user kmarko By Kurt Marko December 18, 2020
Drilling down into the implications of the SolarWinds incident - lessons to be learned...again.


Few areas of IT seem more futile than cybersecurity, where incessant angst and endless budget increases haven’t made businesses and government agencies any less vulnerable to the kind of attack that prompted an ​urgent Sunday night order by the US CISA​ that all federal agencies “disconnect or power down” a widely-used software product.

One would hope that spending a quarter ​billion dollars per day​, up 34% in three years, worldwide on new products, services and consulting would prevent attacks that could compromise 85% of the Fortune 500 and dozens of agencies, but apparently not.

Unfortunately, this week’s revelation is merely the latest in a series of attacks that exploit the soft underbelly that is the software and service providers every organization uses in their daily operations.


Solarwinds is the latest in a long series of supply-chain intrusions

The latest avenue of cyber intrusions is a mundane piece of IT monitoring and management software from Solarwinds, an Austin company that lost $1.75 billion in market capitalization in the hours after ​its software was revealed to be the source​ of an advanced persistent threat (APT) attack that inexplicably took nine months to detect. For those not following the news, here are the key points:\

  • On December 8th, ​FireEye CEO Kevin Mandia revealed that​ the company was attacked by a “highly sophisticated threat actor, one whose discipline, operational security, and techniques lead us to believe it was a state-sponsored attack.” Based on reverse-engineering the techniques, Mandia surmised that the attacker specifically targeted FireEye and went after its Red Team assessment tools used for penetration tests of FireEye customers.
  • In a ​December 13th update, Mandia identified​ the attack as part of “a global campaign that introduces a compromise into the networks of public and private organizations through the software supply chain. This compromise is delivered through updates to a widely-used IT infrastructure management software—the Orion network monitoring product from SolarWinds.” FireEye also “identified multiple organizations where we see indications of compromise dating back to the Spring of 2020.”
  • Later on the 13th, ​FireEye detailed the steps used by the attacker​ to plant malware granting entry into an organization and subsequent steps to proliferate, establish a command and control network, download additional tools, evade detection and steal data. The initial source is a modified version of a SolarWinds DLL that contains a backdoor and is installed as part of routine software updates from a Solarwinds system.
  • The US CISA released its emergency bulletin after ​a Reuters story​ focused worldwide attention on the attack and the Solarwinds vulnerability.

Solarwinds issued a security advisory​ (since updated) specifying the affected software versions and the availability of a hot fix that closes the hole. In an ​SEC filing on December 14th, Solarwinds​ detailed the enormous blast radius of the malware. (​emphasis added)​:

SolarWinds currently believes the actual number of customers that may have had an installation of the O​ rion products that contained this vulnerability to be fewer than 18,000​. The communication to these customers contained mitigation steps, including making available a hot fix update to address this vulnerability in part and additional measures that customers could take to help secure their environments. SolarWinds is also preparing a second hotf ix update to further address the vulnerability, which SolarWinds currently expects to release on or prior to December 15, 2020. ​For the nine months ended September 30, 2020, total revenue from the Orion products across all customers, including those who may have had an installation of the Orion products that contained this vulnerability, was approximately $343 million, or approximately 45% of total revenue.​

  • Unfortunately for Solarwinds’ customers, Orion is only the Trojan horse providing initial entry. Once inside a network, the attackers download additional penetration, surveillance, obfuscation and encryption tools. Thus, merely patching their Orion software only eliminates ​future​ intrusions, but does nothing to eradicate a previous intrusion. Breached organizations also must monitor for indicators of compromise, as ​detailed by FireEye​.
  • On December 16th, ​researchers at a Chinese security vendor announced ​that by decoding the attacks’ communications to a command and control domain, it “discovered nearly a hundred domains suspected to be attacked,” including several tech giants. As ​reported by security researcher Brian Krebs, Microsoft​, with help from registrar GoDaddy, took control of the domain to block the attackers from communicating with compromised systems.


Not the first supply chain attack

The SolarWinds hole is the latest in a long string of cyber attacks that exploit weaknesses in an organization’s software suppliers and service providers. Like the SolarWinds incident, many of these are insidious since they infect a software update or hardware device with malware that activates once inside a target’s network.

I first detailed a hardware-based APT in 2014 that spread via malware inserted into the Windows Embedded OS used by some barcode readers before shipment from their Chinese manufacturer. The malware targeted a zero-day vulnerability in a popular ERP package and, once activated, followed a typical APT pattern of distribution (finding new targets), escalation (downloading new hacking tools), communication (setting up a C2 network), obfuscation (covering its tracks and evading detection) and exfiltration (downloading sensitive data). Since then, cheap connected devices like IP cameras, thermostats and printers have become fertile ground for hackers to launch attacks by breaching obsolete firmware with known holes or weak security configurations.

Two years ago, we had the disputed and weakly-sourced story of servers being compromised by malware embedded in one of the motherboard chips. As I detailed here, the particulars were roundly denied by the named parties, however, the chaotic and corruptible nature of foreign supply chains means that similar exploits are entirely plausible. Indeed, a primary reason the cloud vendors insist upon motherboards with TPM, often of their own design, is to thwart hardware- and firmware-level vulnerabilities.

Attacks can also come through service providers or contractors by exploiting weaknesses in their system security or lax security hygiene by employees. These use a supplier’s access privileges as a beachhead on internal networks and systems from which attackers execute the standard APT escalation playbook. The 2013 hack of Target’s PoS systems is a prime example of the profound damage that can arise from seemingly innocuous sources like HVAC contractors or third-party support personnel.

The WFH era offers hackers another opening for enterprise security breaches by attacking the often-lax security of home networks and PCs. The risk escalates when multiple members of the same household share a wireless LAN, allowing hackers to laterally move from, say, a student’s laptop to their parent’s PC and then to a corporate network.

My take

The SolarWinds incident is the latest in the sorry history of IT security failures, even as corporations have spent the last half-decade pouring hundreds of billions of dollars into a profusion of security products and services. It’s easy to cynically view the entire security-industrial complex as a giant money-making scam, particularly when insiders unload $315 million in stock days before its culpability in spreading malware became public.

Maybe the problem is intractable. More likely is that business and IT executives have confused spending with progress. They would be wise to learn some leadership lessons from the hardwood where legendary Coach John Wooden cautioned players to “never mistake activity for achievement,” since the results show a lot of activity with little to show for it.