Apple is about to learn the painful truth of the proverb, "The road to hell is paved with good intentions" if it proceeds with plans for "Expanded Protections for Children." It's almost impossible to argue with the motives behind the new features "coming later this year, in updates to iOS 15, iPadOS 15, watchOS 8 and macOS Monterey" (macOS 12), however, the implementation has already caused outrage among privacy advocates, security experts and competing service providers.
Apple wants to "help protect children from predators who use communication tools to recruit and exploit them, and limit the spread of Child Sexual Abuse Material (CSAM)" by making its products less desirable for those producing and distributing child pornography. It's a worthy goal since technology, through the proliferation of high-quality cameras in phones and the Internet, has facilitated an explosion in such disgusting material.
For example, the US National Center for Missing and Exploited Children’s Child Victim Identification Program has recorded almost a nine-fold increase in the number of reported images and videos in the past decade. Meanwhile, transparency reports from the seven largest social networks show that the number of content and account removals child abuse and safety quadrupled between 2018 and 2020. Undoubtedly, much of this material was created, distributed and consumed on phones and laptops, many of them, Apple products.
Apple will use this year's annual OS updates — iOS 15, iPadOS 15 and MacOS 12 (Monterey) — to introduce three types of child safety features, two of which unleashed the wrath of the privacy and cybersecurity communities since they use on-device machine learning (ML) to all analyze content passing through a device. The (relatively) uncontroversial additions are changes to Siri and Search that provide background information, for example, links to reporting sites, “when users perform searches for queries related to CSAM.”
In contrast, Apple’s changes to Messages and Photos are considerably more obtrusive since both on-device ML to analyze incoming data and detect sensitive or inappropriate content. Apple has designed its system to perform such analysis without disclosing the content of messages or photos to Apple or third parties, but privacy skeptics and security experts have three areas of concern:
- Fully validating Apple's claims takes time and considerable expertise in advanced cryptography.
- Likewise, checking for back doors or vulnerabilities requires time and effort from a wide range of hackers and security experts. Indeed, many exploits won’t be discovered until after the features are publicly released.
- Distrust about future uses of the technology, namely now it might be repurposed by government mandates to screen for whatever content a particular regime or regulatory body deems illegal or inappropriate. Such repurposing would be particularly dangerous for human rights organizations, political dissidents or other ‘heretics’ from the established social, cultural or scientific order.
Communications safety in Messages is a relatively straightforward application of on-device ML in which all images to or from a protected account (as set up in Family Sharing) are scanned for “sexually explicit content.” According to Apple’s FAQ on the new features:
When a child account sends or receives sexually explicit images, the photo will be blurred and the child will be warned, presented with helpful resources, and reassured it is okay if they do not want to view or send the photo. As an additional precaution, young children can also be told that, to make sure they are safe, their parents will get a message if they do view it.
Apple does not explain how it analyzes the images “while keeping communications unreadable by Apple,” but it likely employs similar techniques to those used in ARKit and the Camera app for face and scene detection. In contrast, its most controversial new feature, CSAM detection, uses a novel photo-matching technique.
CSAM detection is a sophisticated system that involves scanning all photos before they are uploaded to iCloud Photos, looking for matches to images in a database maintained by the National Center for Missing and Exploited Children (NCMEC), an organization that closely works with Federal and international (e.g. INTERPOL) law enforcement agencies to combat child exploitation. Apple’s downplaying the intrusiveness of this feature as applying only to photos stored in iCloud is disingenuous since by default iOS and iPadOS (and macOS devices logged into an iCloud account) automatically store all photos in iCloud. Nonetheless, the privacy protections designed into the system are impressive.
Apple provides a detailed technical description of its CSAM system in this whitepaper, but in brief,
- Apple transforms the NCMEC images into a database of unique numbers using a hashing technology called NeuralHash. The algorithm works well for images since nearly identical photos, for example slightly cropped, transcoded to different quality and color levels (e.g. color to grayscale), generate the same hash value. The hashed NCMEC database is stored on-device.
- Before storing a new image in iCloud (which recall happens automatically behind the scenes for most users), the device uses the same NeuralHash technique to convert the image into a number.
- The system uses another cryptography technique called Private set intersection (PSI) to determine if there is a match between the new image and the restricted database “without learning anything about image hashes that do not match” According to Apple’s technical paper, “The PSI protocol ensures that Apple learns the image hashes in the intersection of the two sets, but learns nothing about image hashes outside the intersection.”
- When there is a match, the device creates a “cryptographic safety voucher” that encodes the match result, encrypts the NeuralHash value and “a visual derivative.” The voucher is then uploaded to iCloud Photos along with the matching image.
- Apple uses the Private Set Instruction (PSI) protocol to share matching images while ensuring that it “learns nothing about image hashes outside the intersection” (i.e. matches to the NCMEC database). The protocol is complicated since it requires that “the device doesn’t learn about the result of the match” before storing an image in iCloud.
- The system uses a technique called threshold secret sharing to protect image information when the number of matches falls below Apple’s warning criteria, but allows it to reconstruct the contents of matching images when the number exceeds its threshold. Apple’s technical paper reiterates that “Nothing is ever revealed about non-matching images during any step of the CSAM Detection process.” The following illustrates this thresholding process.
Apple provides statements from three cryptography experts endorsing its CSAM system and two ostensibly independent papers analyzing the security of its PSI protocol, but these weren’t enough to mollify critics.
Criticism of Apple’s proposal was led by the Electronic Frontier Foundation (EFF) and famous NSA whistleblower Edward Snowden who summed up the slippery slope argument this way, “if they can scan for kiddie porn today, they can scan for anything tomorrow.”
The EFF was more systematic in its blog Apple's Plan to "Think Different" About Encryption Opens a Backdoor to Your Private Life, but echoes Snowden’s logic (emphasis added):
We’ve said it before, and we’ll say it again now: it’s impossible to build a client-side scanning system that can only be used for sexually explicit images sent or received by children. As a consequence, even a well-intentioned effort to build such a system will break key promises of the messenger’s encryption itself and open the door to broader abuses.
All it would take to widen the narrow backdoor that Apple is building is an expansion of the machine learning parameters to look for additional types of content, or a tweak of the configuration flags to scan, not just children’s, but anyone’s accounts. That’s not a slippery slope; that’s a fully built system just waiting for external pressure to make the slightest change.
EFF also objects to other aspects of the changes, including:
- Scanning images in Messages breaks Apple’s promise of end-to-end encryption by passing image attachments through an ML classifier before being displayed.
- ML image classification algorithms are prone to false positives and difficult to audit.
- Apple could easily and furtively change the classifier to censor other types of content.
- The NCMEC database isn’t independently audited nor viewable by users, even though it is distributed to every Apple device as part of the OS.
The head of competitor WhatsApp also speculated that the CSAM system could be exploited by spyware for unknown nefarious activities. A Johns Hopkins cryptographer and the COO of Corellium, Matt Tait, also discussed the possibility of the U.S. DoJ or other law enforcement agencies coercing the NCMEC into poisoning the CSAM database with non-child-porn to capture users doing other ‘objectionable’ (to the agency) things. Tait argued that such efforts would probably get caught by Apple’s manual review process which allows it to decrypt questionable images after an account meets the trigger threshold via threshold secret sharing, a conclusion later confirmed by Erik Neuenschwander, Apple’s head of Privacy, in an interview.
Neuenschwander argued three aspects of the CSAM system make it resistant to governments and other agencies trying to target particular images or individuals.
- The hash list of banned images is built into the OS, thus, shared among all users and not easily or unobtrusively updated.
- The system isn’t triggered until an undisclosed number of targeted images are identified. Thus, it can’t be used to find a few photos, for example, photos taken at an unapproved (by the government) event.
- After the trigger threshold is reached, the system has a manual review process in which an Apple team decrypts flagged images and verifies their content before referring it to law enforcement and/or the NCMEC.
Neuenschwander concludes that (emphasis added):
The hypothetical requires jumping over a lot of hoops, including having Apple change its internal process to refer material that is not illegal, like known CSAM and that we don’t believe that there’s a basis on which people will be able to make that request in the US. And the last point that I would just add is that it does still preserve user choice, if a user does not like this kind of functionality, they can choose not to use iCloud Photos and if iCloud Photos is not enabled no part of the system is functional.
No one wants to protect the moral degenerates producing and consuming the typ of material targeted by Apple’s CSAM system, but it’s worth putting Neuenschwander’s last point in context. Apple’s approach jumps through hoops to protect the privacy of iCloud users by doing most of the processing on-device and anonymized. In contrast, most CSAM systems, such as this tool from Cloudflare or Google’s approach on its Search and YouTube platforms, scan material on the server where the poster’s identity is already known and there is no privacy protection in case of false positives.
Apple correctly chose to maximize user privacy by using the ample computational resources now available on its devices to perform content analysis locally. Indeed, the CSAM system is elegant in its complexity and could serve as the basis for end-to-end encryption of other iCloud content in the future while providing a mechanism for the legitimate review of illegal activity. The slippery slope argument is legitimate, but given its track record, Apple is more trustworthy than most tech companies that often cave to government pressure. A critical element — and perhaps a fatal bottleneck — in Apple’s system is the manual review process.
I agree with John Gruber, that if Apple is overwhelmed by the amount of content meeting its trigger criteria, it could easily end up misreporting innocent photos:
If the number is large, it seems like one innocent needle in a veritable haystack of actual CSAM collections might be harder for Apple’s human reviewers to notice.
Apple is guaranteed to offend someone in attempting to satisfy conflicting goals, but they are both worthy objectives. Prepare to be outraged and see some fine-tuning as flaws in Apple’s system are revealed and fixed.