Planning AI projects

Original article can be found here (source): Artificial Intelligence on Medium

What differs AI projects from other software engineering ventures

Photo by Kelly Sikkema on Unsplash

What made me write this article?

I’m a kind of hipster guy, as I studied ML before it was even cool. I did my master thesis related to sign-language recognition, which resulted in an application able to recognize elementary sequences of the signs. Then I moved to the industry, where I worked in multiple software development projects.

Around 2015, I had a great chance to take part in the AI revolution as deep learning started knocking the doors of commercial software. AI changed the ways we solve our problems, but at the same time, the ways we plan and manage our projects didn’t change.

Isn’t building software all about coding something up? Why would one need to manage AI projects differently?

As an AI Technical Leader, I answer the above questions over and over again when talking to clients, investors, and project managers. Let’s have a look at my explanations once again.

Two types of engineering projects

Firstly, let me put this article in some context and define two different kinds of software projects to provide you with an idea of what I’m talking about.

Let the first project be about building a web app, for example, an online shop. There are different buttons, forms, layouts, and an underlying database. The other one is a deep learning product that recognizes dog breeds once presented a dog image. It should make at least 90% of the predictions correctly. There are mainly raw numbers and interactions between them.

Estimates and deliverables in product development

Experienced software engineers can provide a quite accurate estimate regarding the time necessary to deliver each part of the web application. They have performed similar tasks multiple times already so that they can define deliverables and time range with relatively low uncertainty.

Experienced AI engineers have also implemented and trained many machine learning models that result in 90% of correct predictions. Let’s assume, the last model they did is one recognizing all cat breeds. It took them two weeks to make it running. But, if you ask the engineers how much time they need to build a similar product recognizing correctly 90% of dog breeds, they would struggle to answer. It may take a day, two weeks, two months, or the whole eternity. Moreover, they won’t be sure if they can achieve 90% of correct answers at all.

Such significant discrepancies in estimates do not appear in web apps development. If adding a button or a new form usually takes between a week or two, it will probably never take two months.

Similarly, if there is a requirement for an application to have a button in a specific place of the window, you may safely assume it will have it when deployed.

The process of engineering an AI solution

Building a web application requires implementing numerous features such as buttons or forms. The method of development might be iterative. A product owner selects several features to implement for each iteration, and engineers do their best to deliver them within a given time range. The process repeats several times until all components are ready, and eventually, the app might be deployed.

A deep learning model is made of millions of numbers interacting with each other, and engineers or researchers can define and implement those interactions. The interactions are called model architecture. When the model sees a dog image, it multiples the image by the numbers, and returns a prediction regarding the breed of the dog.

The initial values of the numbers are random. Therefore, the model is not able to recognize anything correctly and it needs to be trained. Technically speaking, the numbers must be adjusted. The training is very similar to a teacher who presents pictures of dogs to his student and explains what breed the dog is. In our case, the training algorithm takes on the role of the teacher, and the model becomes the student.

Moreover, there are various meta-parameters that one can tweak to control the training algorithm, resulting in different rates of correct recognition.

The process of defining and implementing the architecture, tweaking meta-parameters, and training needs to be performed multiple times to reach the desired model accuracy.

Why are AI projects hard to estimate?

Each machine learning project is entirely different. One may say each web app project is entirely different too, and that would be true. However, those are other types of differences.

First of all, a machine learning project requires data for the training algorithm, and data quality highly determines the quality of the final solution. Therefore, it’s impossible to provide estimates regarding time and deliverables, even without having a brief overview of the data. In our example, the data are images of dogs.

If there were only pictures of dalmatian dogs in the data, the model would never see an example of any other breed. Therefore, it wouldn’t know how different kinds of dogs look like. Obtaining 90% of correct predictions would be impossible in such a case, as each dog would always be dalmatian according to the model.

Secondly, one may have a perfect architecture that worked well for a similar task (recognizing cats) but doesn’t work equally well for new data (dogs). In the case of the web app project, the above would mean debugging. But imagine, debugging interactions between millions of numbers when modifying a single value affects others.

Instead, engineers and researchers run experiments examining different architectures and meta-parameters. The experiments are not a random search if performed by experienced professionals. Rather than that, the results of each trial are evaluated and interpreted, which helps in determining the next steps. Therefore detailed planning for a long time ahead doesn’t make much sense, as a single iteration strictly depends on the results of the previous one.

What makes AI project successful?

As you can see, performing estimates without examining the data and planning for a long time ahead is not what you want in your AI project. Instead, you should start with a feasibility study and perform short iterations. Moreover, your engineers need to work with business closely.

A feasibility study helps in setting up reasonable expectations and timelines towards final goals. Data analysis and data quality assessment provide your engineers with a good understanding of the problem and what machine learning algorithms are applicable. It may also save you from failed investments if the engineers determine the problem can’t be solved with the current state of the art, or due to data quality.

In principle, CRISP-DM might define the general AI project methodology. In parallel, Kanban works well in tracking the project and planning the next actions effectively. Combining those two makes a quite robust framework.

Process diagram showing the relationships between the different phases of CRISP-DM (CC BY-SA 3.0)

Clear project metrics that are understandable both by engineers and business help in ensuring the development goes in the right direction. Weekly evaluation of the metrics by both business and technical representatives guarantees a common understanding of the goals and priorities.

Summary

I hope this article has convinced you that AI projects are slightly different from any other software engineering venture. Therefore, they require a distinct approach to estimates and deliverables.

It would be great if I could hear back from you and meet your experiences in AI projects and what works best for you. I am also happy to answer further questions. Therefore don’t hesitate to comment or drop me a line!