It’s not often that one's source of IT equipment becomes a foreign policy football, but that’s the situation when it comes to networking gear from Chinese firms, notably Huawei and ZTE.
Both are facing an executive order that would ban their sales in the U.S. due to potential security and privacy risks of equipment from companies that have provided the IT foundation for China's surveillance state.
The concerns center on network switches, routers and cellular base station transceivers that form the backbone of today's data and communications services. So far, the evidence against Huawei, which is the primary target, deploying equipment that can surreptitiously exfiltrate data to its PRC masters is somewhere between non-existent and circumstantial, and its founder and CEO vehemently denies the charges.
However, even the potential for backdoor access to critical routers and switches in telecom or enterprise networks is enough to spook business executives, particularly when there is alternative equipment available from more trusted vendors.
The data or lack thereof is on Huawei’s side since its equipment hasn’t been shown, at least publicly, to be compromised, even amidst persistent charges of hidden threats. Indeed, stories of Chinese hardware backdoors, like Bloomberg's reckless, high-profile accusations about compromised servers sold to major cloud providers have been debunked (see my coverage here). Nevertheless, there are numerous examples of apps, many sourced from China, acting as Trojan Horses for data theft.
Pair that with Chinese-manufactured IoT devices left wide open (whether intentionally or through carelessness) for attack and infection and at least one incident where IoT hardware was deliberately planted with malware used to penetrate private networks and distribute software in a complicated system for data theft and there are reasons to be leery of Chinese hardware.
Unfortunately, Huawei is learning a lesson anyone accused of a crime where there’s no direct evidence supporting the charge, it’s almost impossible to prove a negative.
Distrust, but verify
In cases where two sides to an agreement distrust each other, active, continual verification is the best course of action, which is the goal of U.K.’s National Cyber Security Center (NCSC) when it comes to Huawei network equipment. The agency created a Huawei Cyber Security Evaluation Centre (HCSEC) Oversight Board in late 2010 to assess and mitigate risks of Huawei equipment in the U.K.'s critical infrastructure. In its fourth annual report released last July, the board summarized its findings and concerns regarding Huawei’s software quality, security and Huawei’s development processes.
The HCSEC audit found no glaring security holes, but problems with poor software quality, record keeping and quality auditing, but concluded that Huawei's U.K. operations "ability to operate independently of Huawei HQ has been completed, with – again – no high or medium priority findings." The report did identify "shortcomings in Huawei’s engineering processes [that] have exposed new risks in the U.K. telecommunication networks and long-term challenges in mitigation and management." However, HCSEC found nothing that would support a ban of the type proposed by U.S. officials.
The HCSEC report provides critical context to a recent leak from the NCSC as first reported by the Financial Times exposing a widening rift between the U.S. and U.K. regarding the security and reliability of Huawei telecommunications equipment. According to the FT:
The U.K. National Cyber Security Centre has determined that there are ways to limit the risks from using Huawei in future 5G ultra-fast networks, two people familiar with the conclusion, which has not been made public.
Reuters later independently confirmed the information, writing (emphasis added):
'We don’t favour a complete ban. It’s not that simple,' one of the sources told Reuters on Monday after a Financial Times report on Sunday said that Britain had decided it could mitigate the risks of using Huawei equipment in 5G networks.
Reuters adds (emphasis added):
Two sources said the NCSC did not think it was necessary to completely bar Huawei from British networks, believing it could continue to manage any risks by testing the products at a special laboratory overseen by intelligence officials.
Both sources, who spoke on condition of anonymity because of the sensitivity of the matter, said the position was consistent with public statements made by the NCSC and British officials.
Which obviously refers to the July HCSEC report. After the FT report broke, the NCSC attempted to add nuance to its Huawei stance. In a conference call Wednesday, Ian Levy, technical director of the NCSC noted ongoing problems with Huawei equipment. As reported by the Washington Post, Levy said:
Last year we said we found some worrying engineering issues. As of today, we have not seen a credible plan [to address the issue]. That’s the reality of the situation, unfortunately.
Later on the call, NCSC CEO Ciaran Martin added:
I would be obliged to report if there was evidence of malevolence [by Huawei] and we have yet to have to do that.
The U.K.'s position on Huawei is providing fodder for other European nations to soften their own stances on an outright ban, since a day after the NCSC leak, Germany appears to be ready to allow Huawei on its infrastructure. As reported by the Wall Street Journal:
The German government is leaning toward letting Huawei Technologies Co. participate in building the nation’s high-speed internet infrastructure, several German officials said, the latest sign of ambivalence among U.S. allies over Washington’s push to ostracize the Chinese tech giant as a national security risk.
Critically, the Journal reports that like the U.K., German cybersecurity experts haven’t found evidence of Huawei subterfuge or backdoors in its equipment exploitable by Chinese intelligence agencies. Quoting the article (emphasis added):
A recent probe by Germany’s cybersecurity agency with help from the U.S. and other allies failed to show that Huawei could use its equipment to clandestinely siphon off data, according to senior agency and other government officials. An official at the Federal Office for Information Security, known as BSI, and two cybersecurity experts at the interior and foreign ministries said probes in allied countries in addition to Germany hadn’t uncovered any indication of wrongdoing by the Chinese company.
The HCSEC report spends more time identifying problems of software quality and reproducibility than security. Of course, as network consultant and analyst Greg Ferro correctly observes:
Software with bugs is also more likely to have security problems. If the developers aren’t using good code practices or adopting best practices then its more likely some bugs will have security ramifications. Don’t always assume malicious activity.
Ferro adds that there’s a disturbing use of code obfuscation techniques in Huawei software:
It’s been observed that some of Huawei’s products have been put through binary packers which is a bit of a head scratcher but making it difficult to reverse engineer and inspect for back doors. Other network companies do not do this and thus are often more easily trusted.
Such attempts to evade transparency are precisely the types of practices the HCSEC is designed to address and should be addressed in future reports.
Given the lack of demonstrated security backdoors and in light of Huawei’s broad product portfolio and its financial appeal, the HCSEC approach of distrust, but verify is a more rational and equitable strategy than an outright ban. If U.S. cyber experts have information to the contrary, they need to be sharing it and making a more convincing case to both allies and domestic telcos than we’ve seen so far.
Huawei has permeated networks around the world by following the standard playbook of sustaining technology innovators: provide incremental functional improvements at a price consistently below that of incumbent suppliers. While one can question Huawei’s motives and whether its providing loss leaders are merely the seeds of some grand cyber spying plot, the reality is that Huawei equipment has enabled many smaller and less endowed telecom companies, both in the U.S. and the developing world to upgrade their networks faster than they could have using more established sources.
For example, the Wall Street Journal detailed how many rural U.S. carriers rely upon Huawei equipment and are alarmed by the prospect of its being banned. The perspective of the CEO at a South Dakota carrier is instructive (emphasis added):
James Groft, chief executive of James Valley Telecommunications, said replacing Huawei equipment in his small South Dakota network would cost about $10 million and tie up many of his 50 employees. ‘For a period of one or two years, we’ll have to focus on replacing Huawei and not do anything else,’ he said.
‘I’ve never seen anything publicly that Huawei has done anything wrong,’ Mr. Groft said. ‘I would feel better about this if they [the federal government] had assurances there is something credible, and not fearmongering.’
While it’s easy to question why someone would use Huawei equipment when there are more trustworthy and reliable domestic sources, it’s much harder and more consequential than an academic exercise when you’re operating a small, cash-strapped business trying to stay competitive with much larger competitors that are aggressively upgrading their technology and services.
Despite the apparent rigor of the ongoing NCSC Huawei audit, it’s unclear whether the strategy and more importantly, any external technical review, no matter how thorough, is an effective security precaution. I agree with the caveat express by James Lewis, a cybersecurity expert at the Center for Strategic and International Studies, in the FT article:
The disagreement is on whether the U.K. approach (which others will copy) to controlling the risk of using Huawei will work. A public ban might be too much for countries that fear repercussions in China, but the U.K. approach has a little tint of rose-coloured glasses to it.
Further complicating the auditing strategy is Huawei’s practice of heavily customizing equipment during installation. In a CSIS panel discussion on 5G risks and security Travis Russell, director of cybersecurity for Oracle communications summarized the auditor’s conundrum (emphasis added):
So NCSC started this exercise. And a few months ago, they actually sent a communication to all the operators that said: You know what? What we’ve discovered is that Huawei doesn’t build product and then ship it to all of their customers. They send servers to the site and then they send hundreds of engineers to go custom-develop the product on location. So that means that you can’t test the product that’s sitting in the network until you go to that – each specific product and test it while it’s in the network. And that’s just not feasible. And so that’s what they – actually, in their recommendation, they went back to the operators in the U.K. and said: Based upon this development model, you cannot test and make this secure.
It will be interesting to see how Huawei and the various telecom buyers around the world react to the security imbroglio surrounding the company at the upcoming Mobile World Congress. While Huawei will undoubtedly defend its integrity and woo carriers in underdeveloped countries, it could face significant blowback from U.S. and European competitors and regulators.
Despite the best efforts of the U.K. NCSC, I continue to believe that the risks of Huawei equipment don’t warrant the financial benefits for enterprise IT and ISPs that don’t have the resources or time to undertake a continual program of deep, code- and hardware-level technical audits, much less investigate the many smaller Chinese companies in Huawei’s supply chain.