The last mile in AI deployment - where the biggest risks (and payoffs) happen

The industry is replete with information about improving the effectiveness and performance of artificial intelligence. However, the greater risk to organizations is implementing AI to solve enterprise issues and achieve measurable business value.

This is often referred to as the  "last mile" of implementation. Because this term is derived from telecommunications and the actual physical delivery of goods and services, it may not be the best choice of language, but it's already in everyday use. The challenging issues of making technically, high-performing systems requires excellent planning and care.

Realistically, ML development is an iterative development and testing effort, not in isolation, but in the context of its eventual application. Moreover, AI models such as machine learning applications co-exist in your digital ecosystem, and their effects require understanding.

Enrico Coiera, MBBS, Ph.D. in In Where Artificial Intelligence Meets Reality, uses this diagram.

In the image, algorithm development is the "first mile," where the feasibly of acquiring the needed data is addressed. Data is subsequently obtained, possibly labeled, and preprocessed or "cleaned (though the diagram does not depict the level of effort, which is substantial). The middle mile of AI involves engineering, developing and testing different algorithms, features, and operational performance. These algorithms were established in actual processes, evaluated on known issues in the last mile.

Coiera's diagram is a simplified model of the three miles of AI. In reality, it is more complicated. For example, the first mile can be very time-consuming, and each "mile" has its complexities. For instance, first mile challenges" include gathering and curating high-quality data. A delay or unanticipated complexities in data acquisition often slow down the entire project. The middle mile contains the challenges of data-driven algorithm development, such as identifying and mitigating biases, causal inference, overfitting on training data, and generalizing any models and algorithms developed.

The above diagram is helpful for a high-level look at the problem, but as Greger Ottosson writes in: Solving Machine Learning's ‘Last Mile Problem' for Operational Decisions

In effect, multiple ML models are often used in a business decision, combined with business rules that express policy. Modeling a decision is the art of combining these predictive- and prescriptive assets.

The truth is that Machine Learning is a probabilistic method and not ideally suited to adhere to deterministic policies and rules. Applying business rules after ML-based predictions or classifications will typically deliver both better conformance and transparency (see this figure for an illustration of this process)

In practice, an AI development project has many more steps:

1. Assemble a team; AI isn't a heroic act.
2. It's cliche to say "Get IT and SMEs working together," but some degree of that is necessary.
3. Create a hierarchy of things to solve.
4. Pick one with noticeable business value (generally not a POC).
5. Research the data needed.
6. AI/ML is not one thing. It may involve intelligent automation, content understanding, and predictive analytics, for example.
7. If data is not available or too expensive/time-consuming, go back to 2.
8. Campaign the program to team.
9. Label the data.
10. Train the model.
11. Determine via mathematical tests for bias and fairness (if it applied to the social context).
12. Test, test and test.
13. If you get the results you expected, there is probably no value; change the model parameters.
14. Benchmark the cost of running and maintaining the model.
15. Complexities arise from hybrid architectures, data drift, model drift and the emergent complexities of inserting the model into your existing unfractured.

In the last mile, it should be apparent that AI does not do anything in isolation. It connects to other operations and develops emergent properties that need to be anticipated.

Some last-mile AI challenges:

Measurement, Generalization and Calibration: The first step is to assess how well AI performance completes its assigned task. If it outruns the proposed resource, redesign is called for. The AUC, Area Under the ROC Curve, a classification performance curve, is an easy metric to evaluate, but it's only a start because the actual performance does not assure that a purely technical measurement is sufficient. Evaluating how well the model works is an effort to determine how well it affects people and processes.  

How well a model performs once in production is a function of the consistency of new data with original data. Often, models do not perform well in practice when the official data depicts a different population. 

Local Context: Organizations are a network of people, processes, and technologies, but different areas of an organization may exhibit different platforms, practices and governance. Corporate networks are dynamic. The "fit" of an organizational network changes, and decisioning systems like AI can become dysfunctional. There are other last-mile issues such as empowering users, updating models and even contract management.

The biggest AI deployment impediment for most companies is the last-mile infrastructure for connecting AI into the business. Much is made of the "data problem" or the "skills shortage," but when a model is finally ready for production, there is a significant gap in understanding  the "Last Mile."

As AI development becomes a more automated process, skill in ML lifecycle will increase, and last mile issues will decrease, allowing organizations to focus on the usefulness of the models as an enhancement to the decision-making process. As Enrico Coiera, MBBS, Ph.D., wrote in Where Artificial Intelligence Meets Reality:

AI development should not be seen as a linear journey stretching from the first to the last mile. Instead, an implementation should be seen as an agile, iterative, and lightweight process of obtaining training data, developing algorithms, and crafting these into tools and workflows. Doing so risks the end product not meeting real-world needs, just as it does with software. Finding the right balance between reusing general technology and building to meet local conditions will be crucial. Either way, AI should not be created far from the place it will be used. Ideally, they should be born deep within the network that they will ultimately live in.

My take

The AI last mile may represent the most significant risk to the enterprise. There is complexity in putting AI into production. Perhaps IT principals are mesmerized by the terms "AI" and "AIOps," assuming it's just another case of project management and governance. AI may not only disrupt Integration into existing business processes, it can do so in mysterious ways, and may escape scrutiny.