Interpretability of machine learning models

Source: Deep Learning on Medium

Interpretability of machine learning models

The development of Machine Learning models that process large amounts of data greatly improves the performance of predictions. Nevertheless, these models raise many questions about their interpretability, which can lead to rejection by the business lines or customers using them. Data Scientists who wish to deploy these models must therefore propose a rigorous approach to improve the understanding of the results. Improving the understanding of Machine Learning models is therefore essential for their successful implementation within companies.

On January 17, 1991, Operation Desert Storm began, which pitted a coalition of 35 states, led by the United States, against Iraq. After the first successful air raids, the coalition decided to launch the ground assault. To everyone’s amazement, when the American tanks opened fire, they pointed their guns at their allies and began to pound them, breaking up the coalition in the process.

It is necessary to have a good understanding of Machine Learning models

This episode of political fiction has its origin in the consequences that could have resulted from a misinterpretation of Machine Learning models. Indeed, during the 1990s, the American army tested an automatic enemy tank detection technology based on image recognition learning algorithms. However, within the sample, the most discriminating factor for detecting the presence of an enemy tank was the colour of the sky: photos showing a landscape including an enemy tank were taken in good weather while those without were taken in bad weather. Once the detection models had been calibrated in this way, a simple storm was enough to make an entire coalition wobble… This example highlights the need to have a good understanding of the Machine Learning models in order to be able to use them correctly.

This is all the more true as we live in an era where algorithms are taking an increasingly important place in our daily lives: credit granting, dating sites, choice of routes, etc. However, this multiplication of algorithms raises many questions: how were they built? how do they work? how do they explain their decisions? The answers to these questions are a relatively new but expanding field of research in the scientific world. These questions must be taken seriously by companies wishing to equip themselves with such tools, at the risk of seeing the relationship with their customers and the support of the business lines for data access projects deteriorate.

To illustrate our point, let us take as an example the algorithms used by banks to determine each other’s borrowing capacity. Let’s put ourselves in the shoes of a young asset we will call Charles who is looking for his first real estate investment. Charles goes to the bank to find out the terms of the loan and provides a fairly wide range of his personal data (age, salary, marital status, etc.).

To his great surprise, his bank advisor told him that he was ineligible to take out a loan. The new credit allocation software based on Machine Learning and allowing you to allocate credit to your customers “with just a few clicks and with unparalleled accuracy” has recently been implemented and is expected to use the latest state-of-the-art techniques in Artificial Intelligence. However, the performance of the algorithm was preferred to its interpretability, which leaves the bank advisor in a quandary with this new software since he cannot explain to Charles the factors that discriminated against his file.

How to reconcile performance and interpretability?

This example illustrates the compromise faced by any development project related to Machine Learning: where is the ideal cursor between performance and interpretability? Modeling is generally characterized by an inversely proportional relationship between performance and interpretability.

Thus the universe of Machine Learning models can be divided into models that can be interpreted by nature (multilinear regressions, decision trees, etc.) and so-called “black box” models (random forests, neural networks, etc.) as summarized in the illustration below:

The success of a Machine Learning project within a company is based on the following five golden rules:

i) The model must solve a clearly identified problem corresponding to a business need;

ii) The model must capture reality as well as possible without presenting any overlearning bias and by being generalizable;

iii) The model must be able to be explained and popularized among the trades in order to gain their support;

iv) The model must be adapted to the requirements of the end customer; and

v) The model must meet the requirements of the regulator.

Improving the interpretability of Machine Learning models is one of the main levers available to Data Scientists teams to meet the criteria for successful project development. This makes it possible to avoid the duality of Interpretability vs. Performance, which could work against potentially more efficient models.

Our previous example illustrates the need to have a good understanding of Machine Learning models: the bank advisor is unable to explain the result of the model to the client, who finds himself in the misunderstanding of the bank’s decision, which leads to a deterioration of the pre-existing trust relationship between the bank and its client.

Understanding and explaining models is therefore one of the major challenges in Machine Learning projects. What is the process to be followed to achieve this? What are the existing solutions? To answer these questions, two categories of techniques emerge: global interpretability and local interpretability.

Global interpretability seeks to identify the most important variables of the model, in particular through a careful analysis of the contribution of each variable to the model’s output data. What is their contribution to the model’s performance? What is the relationship between each variable and the output of the model? This must also be complemented by a critical look at the economic meaning of the behaviour of the main variables. The overall interpretability should eventually (i) improve the understanding of the model by business experts and thus (ii) make the results more appropriate.

A method commonly used for global interpretability is the Partial Dependency Plot method, which consists in freezing all variables except the one you want to analyze. Thanks to numerous simulations, it is possible to evaluate the behaviour of this variable in the model. It is then sufficient to apply this methodology to all the variables of the model to be able to know the impact of each variable and on the output.

Reduce the gap between data science and business

In our example, the overall interpretability allows the bank to accurately understand the criteria and variables that the model uses to estimate the credit risk associated with a particular type of profile. This control of risks and models is nowadays essential for European banks, particularly with regard to the regulator, who is increasingly demanding in the calculation of banking risks[1].

Local interpretability, on the other hand, seeks to decipher the behaviour of the model at the level of an individual by identifying the impact and local contribution of each variable. This method should improve the communication and justification of the results of the algorithm to the end user.

One of the methods commonly used for local interpretability is the so-called Shapley Value method, which will highlight the contribution of each variable to the difference between the prediction and the average of the predictions. In Charles’ example, this helps to highlight the strengths and weaknesses of his case. Thus, the bank advisor could explain that age, salary and level of savings were the variables that contributed most, and to what extent, to the final decision made on his file.Projects related to Big Data in companies often lead to the improvement and automation of the operational chain or the fluidification of the customer experience through a simplified and unified experience. Nevertheless, not taking into account the expectations of business, customers and the regulator upstream of project developments can lead to project failure, especially when projects are based on “black boxes”. This is why the interpretability of models offers an essential opportunity to reduce the gap between data science and business. In this respect, devoting part of its efforts to the use of interpretability methods could eventually lead to the acceptance of more efficient models. Interpretability is therefore one of the key factors for the successful implementation of decision algorithms in companies.

by Morand STUDER, Pietro TURATI and Clément TEQUI

[1] Basel III: Finalising post-crisis reforms, Bank for international Settlements, December 2017