Beware the AI risks of over-automation and hyper convenience

Automation, robotics and AI have created much angst in the popular press, among tech elites like Elon Musk and here in diginomica about the long-term implications for individuals and society. The discussion often centers on the displacement of human workers with algorithms and machines, but some, like Musk,  warn of a dystopian future in which intelligent machines go HAL-9000 style rogue and destroy civilization by accident or intent. While these are intriguing topics for economic and philosophical discussion, they’re of a sufficiently abstract nature as to be of little (but non-zero, more later) relevance to those concerned with running a business. Instead, as my colleagues and I have written many times, automated, ‘intelligent’ algorithms and machines offer so many benefits to overall efficiency, process reliability and business insights that those who ignore them doom their organizations to the crushing competition of more progressive competitors.

Nevertheless, turning processes and decisions over to machines can’t be a reflex action since there are often subtle downsides and risks that sometimes outweigh the benefits. I had an epiphany into another way that automation and self-tuning AI algorithms are a double-edged sword after reading Tim Wu's recent column entitled The Tyranny of Convenience. Wu is the Columbia professor and media scholar that coined the term "net neutrality" and whose book, The Master Switch, described how the combination of commercial success and network effects often leads communications, media and Internet companies to evolve into quasi-monopolies operating tightly controlled walled gardens.

While related to that work, his thesis in the column is that one way of enhancing and exploiting such network effects is through a relentless pursuit of customer convenience to which, resistance ultimately is futile. Whether its Amazon with Prime and 1-click ordering that makes it easier to buy everything in one spot rather than shop for the best price or Apple's iTunes store that made it cheap and hassle-free to buy music rather than pilfer it for free from Napster, as Wu says, "convenient beat out free."

True enough, but I think the relevance to what businesses might lose at the altar of convenience is a point he later makes in the context of the individual – putting your life or business on autopilot eliminates the effort, trials and yes, sometimes errors that can lead to self-satisfaction. I contend that it also thwarts the insights and flashes of brilliance that are necessary to innovation. Summarizing it using one of my favorite axioms – You can’t invent the light bulb by continuously refining the candle.

Convenience vs struggle, automation vs innovation

Wu uses the image of human struggle as something that provides meaning to life, as in the popular saying, "the journey is the reward." Unlike 20th Century conveniences that eliminated the physical drudgery of housework or dangers of physically demanding jobs, Wu sees the modern incarnation of convenience as eliminating mental effort to the point of enslavement. As he puts it,

Today’s cult of convenience fails to acknowledge that difficulty is a constitutive feature of human experience. Convenience is all destination and no journey. But climbing a mountain is different from taking the tram to the top, even if you end up at the same place. We are becoming people who care mainly or only about outcomes. We are at risk of making most of our life experiences a series of trolley rides.

He's correct at a psychological or spiritual level, but Wu then pivots into an area that affects every business and office worker (emphasis added).

An unwelcome consequence of living in a world where everything is ‘easy’ is that the only skill that matters is the ability to multitask. At the extreme, we don’t actually do anything; we only arrange what will be done, which is a flimsy basis for a life.

When every process is modeled and cast in algorithms and bureaucracy, every task pre-programmed and every decision narrowed to a multiple-choice set of AI-selected answers, we risk atrophying human initiative, sapping group morale and destroying an organization's collective creative juices. Indeed, one IT consulting firm warns that automation risks dumbing down your workforce by replacing subject matter expertise with algorithms, eroding the understanding of why processes work in a particular way and undermining problem solving skills. Likewise, HR firms are already warning companies to prepare for the deleterious effects of AI, robotics and automation on employee morale since after watching a few viral videos it’s natural for employees to assume that the machines are coming for their jobs.

Automation complacency

Another significant risk of hyper automation is cognitive carelessness when doing tasks that are primarily machine controlled, but require human intervention in particular, often abnormal situations. Indeed, so-called automation complacency has been shown to have disastrous consequences when airline pilots get used to flying on autopilot or drivers become over reliant on vehicle automation and has even been implicated in the recent spate of fatal collisions by U.S. Navy vessels.

I discussed the broader implications of automation complacency in a 2016 column that briefly summarized the work of Nicholas Carr’s book, The Glass Cage. My conclusion then is perhaps more relevant today as we see regular examples of ‘intelligent’ systems performing in ways their designers didn’t intend.

Ceding greater control to software automation comes with a risk. Whether it’s increasingly sophisticated BI systems that can make business decisions or software to automate and accelerate business processes and transactions, business leaders and technical professionals can’t let their skills atrophy in the face of algorithms that usually get it right, but fail in the most spectacular of ways. The consequences of automation complacency are rarely fatal, but as organizations from Wall Street to retailing have discovered,  they can be financially and reputationally ruinous. Caveat Utilitor.

My take

Continued advances in AI, robotics and other programmable systems mean that automation will continue its march through virtually all aspects of life. Whether automation is employed to improve personal convenience or business efficiency, there are downsides that require thoughtful consideration in when and where it's appropriate and how it should be designed and used. Wu cautions about the personal ramifications of uber-convenience that eliminates the satisfaction from a hard job well done. As he puts it,

Struggle is not always a problem. Sometimes struggle is a solution. It can be the solution to the question of who you are.

I believe similar values of the struggle and effort of problem solving accrue to businesses in the forms of out-of-the-box epiphanies and the breakdown of groupthink. Of course, automation should be applied in areas where well-understood rote tasks can be systematically performed to increase quality, consistency and security by eliminating human errors. On this point, I agree with the subject of Jon Reed's column on the self-driving enterprise,

Laluyaux acknowledges we are on the beginning of the cognitive automation journey. Like self-driving cars, the self-driving enterprise will be a process, not a flip of the switch. We’ll gradually turn over more decision automation to intelligent machines, designed with proper human interventions.

I would add however, that the intelligent machines and systems shouldn't just be designed for manual override, but with guardrails that reinforce higher level human skills and impede the skills atrophy that happens with continual reliance on automation. I also believe that organizations should leave ample space for human creativity, insight and epiphanies that only occur in the face of struggling to solve hard problems, since here, convenience is the death of innovation. Edison's wisdom remains true in the age of intelligent machines:

Genius is one percent inspiration, ninety-nine percent perspiration.