Google's March Madness can help demystify machine learning
- Summary:
- What can Google's attempt at predictions during March Madness teach us about the business application of Machine Learning? Quite a lot.
The risk for any such over-hyped technology is that rampant dishonesty and misuse leads many to discount the entire concept as an elaborate scam.
That's not an unreasonable response. In the case of Bitcoin, a 'speculative scam' is an apt description, however, it's an overly cynical reaction to something as broadly useful as ML.
In hopes of redeeming the baby even as we throw out some fetid bathwater, it's time to step back and examine how ML and AI technologies more generally can help organizations answer questions and analyze increasingly abundant data in valuable new ways.
What better example than using ML to predict results in the annual hoopfest that mesmerizes America for three weeks every spring, aka March Madness?
The holy grail of NCAA hoops analysts is picking the perfect bracket: not only the ultimate champion but also the winner of every game. Most of the billions of dollars wagered on the games each year goes towards bracket contests in which contestants pick the winner of every matchup spanning six rounds and 63 games. There are various scoring systems, but in general, the entry with the most correct (or is it least incorrect?) picks wins.
The nature of a single-elimination tournament and the unpredictable nature of human behavior means that a single incorrect pick in an early game snowballs in later rounds. The size of the field means that no one has ever come close to picking a perfect bracket, correctly matching all 63 games.
Warren Buffett is so confident of its impossibility that he has a $1 billion insurance policy waiting to pay out to anyone that achieves perfection. He needn't worry since no one has ever come close to even getting through the first two rounds unscathed.
Between amateurs and experts, countless systems relying on hunch, statistics, team matchups even school mascots have been tried. Until recently, few have tried using ML, but the tournament provides an excellent proving ground for testing and comparing data-driven AI techniques and illustrates how these techniques can be applied to business problems.
Google Cloud and NCAA ML March Madness competition
For the fifth time, Google Cloud and the NCAA teamed to sponsor an ML competition to pick both the Men's and Women's tournament brackets with $100,000 in prize money split between the top three entries in each.
The competition is hosted on Kaggle, a site Google acquired last year, that hosts ML and data analysis competitions and associated discussions to foster technology improvements and sharing of ideas. This year, the entrants not only have access to better ML computational resources, whether on Google Cloud or others, but a vast new dataset comprised of more than 40 million plays from every NCAA Division I Men's and Women's game since 2009.
The hope is that such granular detail will enable smarter, more accurate models that could include compound second- and third-order parameters such as performance in pressure situations or various game situations. An extreme scenario from the Google blog announcing the program illustrates the type of question such data can answer,
How many times has a team with 3 freshmen starters shot better than 50% from three point range and had a 2:1 assist to turnover ratio when they were losing by 8 points with 6 minutes to go against an opponent ranked 5th in three point shots allowed?
While the details of each of the 934 entries aren’t available, Kaggle is updating a leaderboard that scores the accuracy of each. However, unlike the typical office pool bracket contest, the Kaggle entries don’t merely pick winners and losers, but the probability of an outcome since ML models don’t result in categorical predictions, but probabilistic estimates.
Each entry is scored using a popular metric for judging the accuracy of ML models, logarithmic loss, which doesn’t just reward getting the correct result for each game, but factors in one’s confidence in the outcome.
Thus, if a model predicted with 90 percent certainty that number one seed Virginia would beat number 16 seed UMBC, it would be severely penalized given the unprecedented upset that ended up occurring.
In contrast, if a model correctly predicted with 55 percent certainty that number four seed Gonzaga would beat number five Ohio State, it would get less credit than a model that had higher confidence in the same result.
Applying ML to retail, pricing and logistics
Modeling a basketball tournament might seem to have little relevance to business problems, but the same sorts of data selection, normalization and model testing that goes into picking winners of a game, can be applied to many other questions and data sets.
Logistics and supply chain optimization present one example. A shipping company or retailer with multiple warehouses, each with different sets of inventory, can model the vast number of combinations for supplying a given basket of goods to hundreds of customers in different locations, looking for the fastest, least expensive, most efficient ways of scheduling trucks, pickups and deliveries.
More generally, shipping companies can use neural networks to solve the well-known Travelling Salesman Problem for route optimization.
Retailing presents another area in which combinatorial problems with millions of possible solutions can finally be addressed using ML. As I wrote last year, Jet.com, now part of Walmart, solves a type of supply chain management problem by using machine learning to minimize the cost of an entire basket of goods that are provided by different merchants using various distribution centers and shipping charges.
While competitive and temporal data is routinely used for dynamic pricing (think airline tickets), more sophisticated ML applications would allow retailers to adjust the prices on highly correlated baskets of goods to maximize overall profit, not just that for a particular item.
Stores have long used loss leaders to generate traffic that presumably results in greater overall sales of higher margin items. By using ML, they could quantify the optimal items on which to sacrifice margins and thereby generate the most sales of high margins products. For example, around holidays and major sporting events, the best strategy might be to take a hit on paper plates, napkins and barbecue sauce, by attracting party planners, which leads to higher sales of more profitable beer, chips and chicken wings.
As a side note, remember the old Walmart data mining fable about beer and nappies?
Each of these problems could be tackled with the same sorts of massive data sets, whether transaction logs or historical delivery records, that the March Madness entrants are using to select among the 9.2 quintillion (2^63) game possibilities.
My take
Keen observers of the realité of IT like my colleagues here at diginomica are right to be skeptical of AI claims when they hear, as Jon Reed reported, panelists claim that AI is everything from “the ability to imitate human intelligence” to “a common marketing play.”
Dennis Howlett’s cynicism was understandable when he replied, “In other words: ‘AI is whatever the fook you want it to be.’ “ Business and technology leaders must filter out the noise of marketing and self-promotion and look at the concrete results of using AI to answer business questions.
One of the areas all non-specialists must struggle with is the mystery of how ML algorithms arrive at their conclusions. The black box nature of AI leads many to question the validity of results and look for hidden biases.
Researchers are making progress on providing transparency into the inner workings of ML, such as this excellent graphics-rich description by Google, the reality of multi-layer neural networks that slice and dice data along multiple dimensions means they will remain inscrutable to most users. Instead, business execs should focus on the results: how accurately does a model answer particular speculative questions, do its answers make intuitive sense as you change input data, and are predictions smoothly consistent as the inputs slightly change over time. The maxim, "trust, but verify" is inherent in the development of all ML models, but is also a practice AI users should continue.
As I mentioned last week, tech news often focuses on product specs, not business utility. While I expect to see plenty of AI product news next week at the NVIDIA GTC conference, I'll be particularly interested in the applications and real-world results of AI in business, science and medicine. Let me know what you want to see and I'll keep an eye out.