Purported server sabotage illustrates potential risks of Chinese supply chains
- Summary:
- In our second story about the purported Chinese hardware hack, we consider the supply chain issues.
Not only were the implications beyond dire — virtually undetectable hardware inserted at the point of manufacture allowing unfettered access to systems at their most vulnerable points — but the alleged victims were two of the biggest names in tech: Apple and Amazon.
That the Chinese government would use its authoritarian muscle to corrupt an electronics supply chain within its borders is unremarkable. Previous incidents involving IoT devices have been discovered and Chinese telecom manufacturer ZTE has been sanctioned for doing as much. However, the potential scope and ramifications of a wholesale compromise of systems powering cloud data centers and online services around the world would be unprecedented.
What makes the story even stranger is the vehemence, promptness and unequivocal nature of the denials from both Apple and Amazon: no vague weasel words and non-denial denials this time.
The claim
The massive story, which would print to more than 20 pages, ran in Bloomberg on October 4th. It’s worth reading in its entirety, but the story's key points are these:
- Server motherboards made by Supermicro, one of the world's largest manufacturers of such components supplying many well-known brands and online services, were modified during assembly at one of four subcontracting facilities in China. Suspected agents of China's People's Liberation Army (PLA) bribed or threatened plant managers to secure delivery the chips allowing backdoor access and their subsequent assembly on targeted motherboards.
- These components eventually went into systems used by about 30 companies, including Apple, Amazon, an unidentified "major bank" and government (unclear which, but assuming U.S.) contractors.
- The chips, which are said to be the size of a pencil tip (roughly 0.5 mm), are thought to connect to the baseboard management controller (BMC), a specialized component that allows remote system administration such as resetting or power cycling a system and, in some systems modification of BIOS settings. According to the account, the chip's proximity to the system BIOS and memory allowed it to intercept and modify OS code during system boot to create several backdoors, including setting up a connection to a remote system to download additional, presumably more complex instructuctions.
- Compromised motherboards were first discovered after Amazon commissioned an audit of systems and software at Elemental Technologies, a company it eventually acquired and that used Supermicro servers. After Amazon alerted U.S. authorities (unclear exactly which agencies, but the FBI is later mentioned in connection with Apple), subsequent investigations used the chip's "phone home" feature to trace network activity and determine that there were almost 30 companies affected.
- In late 2015, officials in the U.S. DoD called a meeting to brief "several dozen tech executives and investors on the hardware attacks and to request that they develop hardware countermeasures.
Our colleagues at STH have a useful technical discussion of BMC vulnerabilities and some informed speculation of how a vulnerability such as that described might have worked.
The response: Apple and Amazon unload
Bloomberg tells a compelling and cautionary story, but is it true? It cites 17 anonymous sources, more than a year of reporting and more than 100 interviews to back up its claims. In contrast, the two largest 'victims' of the hack, Apple and Amazon, unequivocally and categorically deny it ever happened.
Apple's initial PR response, which given the gravity of the claims was undoubtedly vetted by its senior executives, legal team and security office, notes that it has refuted the claims every time Bloomberg has contacted it over the past year. Apple states (emphasis added),
Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them. We have repeatedly and consistently offered factual responses, on the record, refuting virtually every aspect of Bloomberg’s story relating to Apple.
"On this we can be very clear: Apple has never found malicious chips, 'hardware manipulations' or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement.
(Emphasis added)
Refuting speculation that Apple is hiding something for national security or competitive reasons, the letter later states that Apple is "not under kind of gag order or other confidentiality obligations."
As if this weren't strong enough, over the weekend Apple's VP for Information Security, George Stathakopoulos wrote a letter to the Senate and House commerce committees reiterating its original statement.
According to Reuters, Stathakopoulos writes that
Apple’s proprietary security tools are continuously scanning for precisely this kind of outbound traffic, as it indicates the existence of malware or other malicious activity. Nothing was ever found.
The Reuters article also quotes Apple’s former general counsel as saying that he contacted the FBI’s general counsel last year after hearing of Bloomberg’s allegations. After checking into them, the senior FBI official told Apple’s top lawyer,
Nobody here knows what this story is about.
Amazon was equally forceful in its denial. A posting on the AWS Security blog by AWS CISO Steve Schmidt comes out blazing (emphasis added),
As we shared with Bloomberg BusinessWeek multiple times over the last couple months, this [the claim that AWS discovered malicious chips in Supermicro motherboards] is untrue. At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon systems. Nor have we engaged in an investigation with the government.
"There are so many inaccuracies in this article as it relates to Amazon that they’re hard to count. We will name only a few of them here.
Schmidt goes onto refute specific points in the article including that Amazon discovered modified hardware in Elemental servers and that an audit of AWS's Beijing data center found other hacked servers ultimately causing Amazon to sell the facility to a Chinese firm.
While not the unqualified refutation issued by the companies, the statements of Apple and Amazon were supported, or rather, not contradicted, by both the U.S. Department of Homeland Security and U.K. National Cyber Security Centre, a unit of GCHQ intelligence agency. The U.K. cyber spooks state that,
We are aware of the media reports but at this stage have no reason to doubt the detailed assessments made by AWS and Apple,” adding, "The NCSC engages confidentially with security researchers and urges anybody with credible intelligence about these reports to contact us,
The DHS concurs,
Like our partners in the UK, the National Cyber Security Centre, at this time we have no reason to doubt the statements from the companies named in the story.
My take: The ongoing risk
My colleague Jerry Bowles takes a 'where there's smoke, there's fire' approach to this topic. I'm not 100% behind that argument but I see where he's coming from.
Although the veracity of Bloomberg’s story is in doubt, what’s not is the fact that electronics supply chains running through China are at risk for malware and worse. Lax security on devices manufactured in China allowed hackers to launch one of the largest denial of service attacks in history, leading to the subsequent recall of millions of products.
Worse still was a 2014 incident in which security researchers discovered malware loaded into the firmware of barcode scanners used by many logistics companies. Once the scanners were deployed, the malware, which was a form of advanced persistent threat (APT) scanned the local network looking for Windows systems which they would attack with a then-unknown vulnerability. Once compromised, those systems contacted command and control systems back in China to download more sophisticated payloads designed to extract financial information from ERP systems.
Taken together, past incidents and the Bloomberg report, whether true or not, highlight the vulnerability of equipment manufactured in areas or by subcontractors that are untrustworthy. They also underscore the necessity for companies to both independently verify the integrity of equipment and embedded software and design tamperproof, cryptographically secure hardware security circuitry into their products.
Whether by coincidence or suspicions raised by Bloomberg’s inquiries, Apple, Amazon, Google and others all have developed hardware security chips or circuits for some of their products. For example, Google’s Titan security chip is now used on new servers and network devices in its data centers to establish a root of trust and identify and authenticate access at the hardware level. Likewise, Apple builds a Secure Enclave into the system processors for its iOS devices that secures the bootloader and verifies that system software is cryptographically signed by Apple.
AWS has recently started adding a Nitro security chip to its servers hosting EC2 instances (full details in this re:Invent presentation). Furthermore, as Schmidt stated in his blog post,
Amazon employs stringent security standards across our supply chain – investigating all hardware and software prior to going into production and performing regular security audits internally and with our supply chain partners. We further strengthen our security posture by implementing our own hardware designs for critical components such as processors, servers, storage systems, and networking equipment.
Similar steps will be required of any company sourcing easily sabotaged equipment from contract suppliers that could be compromised. Indeed, China’s new cybersecurity law, which grants authorities broad powers to physically inspect businesses and remotely access corporate networks to check for potential security loopholes could serve as an ideal pretext for industrial espionage and hardware sabotage.
Given the strident denials by the affected parties, I am doubtful that the Bloomberg story is correct as written: there are certainly missing details and misinterpreted clues. Indeed, there might even be some disinformation going on to damage the named companies or provide cover for other nefarious activities as was suggested in Bowles argument.
Like so many espionage and counterintelligence stories these days, we might never unravel the full story. Nevertheless, the scenario outlined is possible; a more sophisticated form of the type of hardware compromises that have already occurred. The lesson for IT equipment buyers is to trust no one and invest in independent forms of hardware security, particularly for equipment intended for data centers hosting an organization’s most critical data and applications.