Why should Machine Learning developers care about transparency?

Photo by Maxime Le Conte des Floris on Unsplash

The modern world is full of algorithms that drive many complex systems and by doing so, affect lives of millions. Not all of them are sophisticated deep neural networks, sometimes it’s just a statistical model or automation process. The common problem is lack of clear communication of how those things work with its developers and often very unaware users. Today, I would like to look at trending approach in the time of hype around AI: end-to-end machine learning models, which are fed with a massive amount of raw data to perform a sophisticated set of tasks of classifications. I will try to consider both the problem they create but also propose ideas for solving that challange.

The first challenging part of end-to-end AI algorithms comes with a restricted co-operation with its developers during the development phase. When algorithm learns entirely by itself, with tons of data using deep learning, model developers could omit one of the most crucial parts of good old machine learning, feature engineering, which till the boom of deep neural networks was probably most important making machines to learn. Giving up this step also creates the danger of polluting the model with biased data, and in consequence indirectly harm people. Detailed examples of this issues are described in “Weapons of Math Destruction” written by Cathy O’Neil.

Applying domain knowledge for construction of better input data for models has two main advantages. First of all, it can enormously improve algorithm performance, especially when we do not have vast amounts of data and from the other. Secondly, it gives meaningful insights about used dataset to model developers. Insights that could be still very useful, even when a model by itself does not meet desired expectations.

After finishing development phase, we need to think what happens when our black box solution finally ships to the production environment, or better to say: human environment. It is even more important to pay attention to safety and reliability, in high-risk industries like a healthcare, transportation or finances decisions and actions made by the AI could affect real people’s health and life. The moment when we delegate a complex task and expect only final results is when we lose insights of what’s happening. The typical example could be an autopilot in a plane, drone or autonomous car which suddenly has a crash and human operators have no clue what was the reason why such system performed specific actions.

In fact, constant interaction and transparency in front of users may prevent a significant percentage of such accidents from happening. A reduction or decomposition of end-to-end models could save many errors when AI systems are used outside of a laboratory or developer environment. Lack of supervision could be required by laws, what’s been already happening in the European Union according to the General Data Protection Regulation. Applications of this regulation will affect not only European based companies but also everybody who wants to use the data of European citizens.

What does the GDPR mean for the AI industry? In specific cases, authors of a model that has been interacting with users need to explain how and based on which rules it has made specific decisions. It may be an answer for a credit request, an automatically calculated insurance cost or adds content presented to a user. As a creator of such systems, we might be asked to prove it’s not biased in a harmful way or doesn’t break any rules of users data protection. Needless to say, developers, project managers, and business owners will need to change their methods for development, in order to adapt.

It all leads to the biggest challenge and opportunity. If we make interpretability of AI systems a universal norm, we could avoid losing trust for this beneficial field of technology or even to the technology itself. I am convinced that by a proper User Experience design, responsible workflows of Machine Learning project and education, we could minimize the risk of AI to arouse fears and benefit from its potential.

Source: Deep Learning on Medium