Over the next two years, Orwell, who was dying slowly of tuberculosis, hammered away at his Remington portable typewriter in an upstairs bedroom of the chilly and damp farm house. The novel was 1984, Orwell’s horrifying imagining of a dystopian freedom-less future in which an authoritarian government monitors every movement and every gesture of its citizens at all times. In this bleak and unfeeling world--devoid of privacy and individualism--giant posters everywhere reminded citizens that Big Brother, the mysterious figure who represented the Party’s power and authority, was literally always watching. Orwell’s protagonist Winston Smith described it like this:
Always the eyes watching you and the voice enveloping you. Asleep or awake, working or eating, indoors or out of doors, in the bath of in bed—no escape. Nothing was your own except the few cubic centimetres inside your skull.
Like all totalitarian states, Orwell’s fictional territory of Oceania sought to control every aspect of its citizens lives through a combination of fear, misinformation, hate, psychological warfare, and surveillance. Citizens who were deemed insufficiently patriotic were labelled enemies of state and vaporized by the Thought Police—their bodies, possessions and personal history erased as if they had never lived.
The technological underpinning of Orwell’s imagined society was the “telescreen,” a kind of two-way television set that watched you as you watched it. Every home and public space had a telescreen and they were never to be turned off. Even the slightest hint of insufficient enthusiasm spotted by the telescreen drew a warning. While protagonist Winston Smith performed his mandatory Physical Jerks exercises one morning, for example, a voice from the telescreen criticized his poor effort.
1984 was published in 1949—13 months before Orwell died at the age of 46.
So widely read and influential has 1984 become that nowadays we would describe it with the adjective “Orwellian.” That is surely a tribute to the author’s understanding of the worst impulses of human nature, the cynical and cyclical nature of political power structures, and his pessimism about the ability of those in power to relinquish it and those not in power to change their predicament.
With the renewed infatuation with authoritarianism now sweeping the globe, it might be useful to remind ourselves that 1984 didn’t happen in 1984 and hasn’t happened since-not because the worst totalitarian impulses were not still there--but because the surveillance technology needed to make it a complete reality didn’t yet exist.
Now it does. Orwell missed the mark by only 40 years. Facial recognition technology, AI, and virtually unlimited computing power makes his nightmare vision of Big Brother not only possible but probably inevitable. That has thoughtful academics, policy and technology leaders concerned.
Last week, the AI Now Institute at New York University, an interdisciplinary research institute dedicated to understanding the social implications of AI technologies, issued its third annual report. Among its key recommendations:
Facial recognition and affect recognition need stringent regulation to protect the public interest.
Such regulation should include national laws that require strong oversight, clear limitations, and public transparency. Communities should have the right to reject the application of these technologies in both public and private contexts. Mere public notice of their use is not sufficient, and there should be a high threshold for any consent, given the dangers of oppressive and continual mass surveillance. Affect recognition deserves particular attention.
Affect recognition is a sub-class of facial recognition that claims to detect things such as personality, inner feelings, mental health, and “worker engagement” based on images or video of faces. These claims are not backed by robust scientific evidence, and are being applied in unethical and irresponsible ways that often recall the pseudosciences of phrenology and physiognomy. Linking affect recognition to hiring, access to insurance, education, and policing creates deeply concerning risks, at both an individual and societal level.
The use of facial recognition technology by a government can encroach on democratic freedoms and human rights. Democracy has always depended on the ability of people to assemble, to meet and talk with each other and even to discuss their views both in private and in public. This in turn relies on the ability of people to move freely and without constant government surveillance.
While noting that there are many governmental uses of facial recognition technology that will protect public safety and promote better services for the public without raising these types of concerns, Smith added:
When combined with ubiquitous cameras and massive computing power and storage in the cloud, a government could use facial recognition technology to enable continuous surveillance of specific individuals. It could follow anyone anywhere, or for that matter, everyone everywhere. It could do this at any time or even all the time. This use of facial recognition technology could unleash mass surveillance on an unprecedented scale.
Smith unveiled six principles that Microsoft have been developing over the past six months that address the concerns the company believes governments need to address as well. These are:
- Fairness - We will work to develop and deploy facial recognition technology in a manner that strives to treat all people fairly.
- Transparency - We will document and clearly communicate the capabilities and limitations of facial recognition technology.
- Accountability - We will encourage and help our customers to deploy facial recognition technology in a manner that ensures an appropriate level of human control for uses that may affect people in consequential ways.
- Non-discrimination - We will prohibit in our terms of service the use of facial recognition technology to engage in unlawful discrimination.
- Notice and consent - We will encourage private sector customers to provide notice and secure consent for the deployment of facial recognition technologies.
- Lawful surveillance. - We will advocate for safeguards for people’s democratic freedoms in law enforcement surveillance scenarios, and will not deploy facial recognition technology in scenarios that we believe will put these freedoms at risk.
Brad Smith ends on a hopeful note:
As George Orwell described in his novel “1984,” one vision of the future would require that citizens must evade government surveillance by finding their way secretly to a blackened room to tap in code with hand signals on each other’s arms – because otherwise cameras and microphones will capture and record their faces, voices and every word. Orwell sketched that vision nearly 70 years ago. Today technology makes that type of future possible.
But, not inevitable.
For the sake of all our futures, let’s hope he is right.