Learn How Your Chatbot Can Detect Irrelevance

Source: Artificial Intelligence on Medium

Automatic Irrelevance Detection

Conversational Interfaces, like chatbots, are usually developed in such a way to only address a very narrow domain. Obviously this is due to time and cost restraints.

Vast amounts of time is spent on handling conversations which are irrelevant to the domain of implementation and the intended purpose of the chatbot.

And as stated earlier, the Out Of Domain (OOD) field is infinite, and effectively defining example utterances are neigh impossible.

Irrelevance Detection Enabled on IBM Watson Assistant

It remains very helpful as irrelevance detection is an element which can assist your chatbot in recognizing scenarios where an user touches topics which are not designed to be addressed.

And address the fact with confidence early in the conversational flow before looping through a few iterations.

Such a feature can help your chatbot to recognize subjects which you did not design and developed for, even if you haven’t explicitly taught it about what to ignore, by marking specific user utterances as irrelevant.

The algorithmic models that help your chatbot understand what the users say are built (ID) from two key pieces of information:

  • Subjects you want the assistant to address. For example, questions about order shipments for an assistant that manages product orders.

You teach your assistant about these subjects by defining intents and providing lots of sample user utterances that articulate the intents so your assistant can recognize these and similar requests as examples of input for it to handle.

  • Subjects you want the assistant to ignore. For example, questions about politics for an assistant that makes pet grooming appointments exclusively.

You teach your assistant about subjects to ignore by marking utterances that discuss subjects which are out of scope for your application as being irrelevant. Such utterances become counterexamples for the model.

Then you need to present counterexamples and often there are false positives where an utterance is erroneously assigned to an entity.

Irrelevance detection is designed in order to navigate any vulnerability there might be in counterexample data as you start your chatbot development.

When you set Irrelevance Detection to enabled, an alternative method for evaluating the relevance of a newly submitted utterance is triggered in addition to the standard method.

To set this on feature to available on IBM Watson Assistant:

  1. From the Skills page, open your skill.
  2. From the skill menu, click Options.
  3. On the Irrelevance detection page, choose Enhanced.

The supplemental method examines the structure of the new utterance and compares it to the structure of the user example utterances in your training data.

This alternate approach helps chatbots that have few or no counterexamples, recognize irrelevant utterances.

Marking User Input as Irrelevant To Build a Counterexample Model

Note that the new method relies on structural information that is based on data from outside your skill.

So, while the new method can be useful as you are starting out, to build a chatbot that provides a more customized experience, you want it to use information from data that is derived from within the application’s domain.

The way to ensure that your assistant does so, is by adding your own counterexamples. If only a few.

Utterances Are Detected As Irrelevant with No False Intent Assignments

In the chatbot development process, you provide example user utterances or sentences which are grouped into distinct topics that someone might ask the assistant about — these are called “intents.”

The bad news…users do not stick to the script.

The chatbot usually gets a variety of unexpected questions that the person building the assistant didn’t plan for it to handle initially.

In these cases an escalation to a human is triggered, or a knowledge base search is triggered. Bot not the most effective response.

How about the chatbot saying, I cannot help you with that.