Can You Foresee Deepest Pitfalls for ROI of AI Projects Using Arithmetic?

Original article was published by Praful Krishna on Artificial Intelligence on Medium


Can You Foresee Deepest Pitfalls for ROI of AI Projects Using Arithmetic?

Alan Turing formulated the Turing Test in 1950. 70 years later data science is yet to pass it. However, in its attempts to do so, especially over the last decade, Artificial Intelligence has now become a household name. Significant efforts by companies like Amazon, Apple, Google, and others have changed everyone’s day to day experiences, as well as our collective imagination.

At enterprises, however, AI is still being met with skepticism. During these 70 years enterprise IT engineers relied on one law — if the system is not accurate, it must be broken. AI comes with an implicit guarantee of less than 100% accuracy. There are other considerations. The costs and benefits of AI projects are not linear. They are not certain, either.

Partly due to this, IT and Business Managers alike have struggled with the ROI of AI projects. They want to consider AI projects like any other — measured in dollars-in vs dollars-out, and rightly so, but in this world two plus two is rarely four.

So I conducted a thought experiment based on my experience with dozens of companies and basic arithmetic.

Let’s start putting things together. Every business makes appropriate investments to get returns. For AI, investments are broadly constant over time — it’s the cost of the data science team or an external vendor. Most vendors price based on time and complexity of solutions. Returns pick up at their own pace and then accelerate. The following graph depicts typical Cost-Reward curves for AI implementation. Each dot is the launch of a new use case. You can download the model from Coseer’s website.

Some powerful truths emerge once we start tinkering around.

Data Scientists Are Not the Bottleneck For ROI

You must hire smart data scientists; they will do a world of good for your organization. Intelligent data scientists accelerate the development process. In our curves, the dots come closer, and the blue line picks up faster. They also help you with the higher applicability of the use cases as well as higher accuracy; all of which brings the break-even closer. Below we compare the curves in the base case and in the case where deployment takes 20% less time due to smarter data scientists. You can see the ROI move.

In my experience, things are rarely as simple. Typically, data scientists are not the bottleneck for deployment, management processes are. In other words, smarter data scientists may not be able to drive ROI unless a larger team matches their brilliance. Data scientists are only a part of the puzzle, and not the biggest one, as you will see below.

There is one more thing — smarter data scientists cost more. Just recently there was an article about how even non-profits are paying north of one million to the best talent.

Accuracy of Your System Is More Important Than You Think

In the world of real enterprises, there is a lot more to the fact that more accurate systems give higher ROI.

An AI system will save you money or time on a process if two things are right: a) the AI system’s use cases cover the particular problem, and b) the problem is diagnosed correctly by the AI. We call these factors applicability and accuracy, respectively. Applicability is driven by a combination of factors typically outside the control of a data science team, but accuracy is more straightforward.

Back to our curves, the more accurate the system, the faster the blue line picks up. It keeps bearing fruit long after our costs are paid. Not only that, the gains compound on each use case. See what happens when the accuracy goes from 60% to 90%.

There is a lot more to accuracy. Accurate systems attract traffic. They provide an incentive for the user not to try alternative methods to get their job done. So, a Virtuous Cycle of Trust picks up — accurate systems reward the user, driving more traffic to the AI system, which trains it better, in turn making the system more accurate.

Data Preparation Costs Can Sink Your ROI by Itself

You can realize substantial gains in your ROI curve if your data scientists can bypass costs of data preparation. IBM vs. MD Anderson episode is the best example to understand this better. IBM’s Watson promised a future where a cognitive computing system will expedite clinical decision-making around the globe and match patients to clinical trials. There was one hiccup — MD Anderson had to first appropriately tag all its oncology data using expensive subject matter experts. It did so, spending $60 million over three years. And then nothing worked. Watson requested further annotation to iterate over their models. That meant further costs with no promises. Anderson walked away, in a much-publicized break-up.

Similarly, for structured data, it is best to use Deep Learning only if hundreds of thousands of records are available. Trying to create data for a model to train properly is the worst possible thing to do.

These kinds of smart decisions about choice of vendor/ technology fundamentally change our curves. See the scenarios with no data preparation costs.

My experience has led to two assumptions in the model: The first thing was to realize that AI projects never work linearly. The outcomes are almost always below expectations initially, and they improve only over time. Every savvy business person dealing with AI takes a long-term outlook across multiple use cases and plans for this uncertainty. This is a key assumption in the model.

Another critical assumption is that the cost of implementing AI is directly proportional to time. The factors involved are essentially the data science team, AI vendors/ consultants, or both. In the AI world, products don’t mean much, solutions do, and they are often priced pro-rated to how complex the problem is.

Your situation is likely to be similar with nuances. I would encourage every business leader embarking on an AI journey to quickly sketch a model like this and foresee the likely pitfalls in terms of ROI.