Your AI Will Embarrass You
Over the course of a single weekend last January, two of the world’s leading AI companies found themselves caught up in embarrassing errors. Facebook’s automatic translation inadvertently translated “President Xi” to “Mr. Shithole” (in Burmese), while Apple drew criticism after Siri responded to queries about the Israel with references to “the Zionist occupation state.”
I don’t envy the relevant PR teams, and obviously Facebook and Apple made mistakes. But the real lesson here isn’t about Silicon Valley bias or antisemitism or cultural sensitivity. It’s about AI. Specifically, don’t use AI unless you can tolerate stunningly embarrassing errors.
As long as AI makes any errors at all, it will make embarrassing errors. That shouldn’t be surprising. To an AI system, actions are either correct or incorrect. When Google Photos labels your friends as “friends”, that’s correct. Everything else is incorrect. And as far as the AI is concerned, everything else is equally incorrect.
Of course, that’s not exactly how the world works. When Google Photos incorrectly labeled two black people as “gorillas”, the consequences were a touch more significant than when it labels a bunch of plantain as “bananas.”
AI designers can, of course, try to work around this. They can give the system 1 point for a correct label, 0 points for an incorrect one, and negative 1000 points for a label that sparks a geopolitical crisis. That may fix the problem at hand, but in the long term it’s a fool’s errand. Crisis-inducing errors are products of context and social nuance that an AI program is unlikely to capture.
Despite AI’s propensity to generate mortifying blunders, it continues to be a tremendously valuable family of technologies. And so, it’s worth reviewing scenarios where it’s successfully deployed.
One obvious example is domains where embarrassing errors are entirely acceptable. Advertising is a great example. As long the advertising platform continues to print obscene amounts of money, embarrassing errors can be shrugged off. When a website shows you an ad for a product that you’ve already purchased, that’s an embarrassing error. Perhaps you’ll screenshot it and tweet to all your followers about how dumb online advertisers are. But that’s as far as it goes. As far as the advertiser is concerned, the only harm is that they showed you an ad for something you’re not going to buy. Put differently, the only problem with an embarrassing error is that it’s an error.
In contexts where embarrassing errors aren’t acceptable, the best approach is to provide a layer of insulation between the model and users. This is what Google ultimately did in the aforementioned case: they implemented a hack that ignores the image recognition model when it produces labels like “gorilla”, “chimpanzee” or “ape.” The crucial point is that they accepted that the model would continue to produce embarrassing errors regardless of attempts to fix it, so they designed an external process to ameliorate those errors.
If the system doesn’t need to be fully automated, one effective approach is to use AI for decision support rather than as a standalone decision maker. This model may be particularly important in healthcare. IBM’s Watson Health reported some promising accuracy metrics, but its tendency to make embarrassing, dangerous errors shredded its credibility with physicians. In one memorable example, it recommended giving drugs that worsen bleeding to a patient with severe bleeding. Had Watson been more explicitly framed as a tool that assists physicians by surfacing treatments for consideration, the errors could have been more tolerable.
There are many open questions about how AI will impact society. The answers to those questions will be shaped by how system designers and consumers handle embarrassing errors. Imagine, for instance, that self-driving cars get to the point where they’re statistically safer than human drivers, but the few fatal crashes are the result of inexplicable malfunctions. Will passengers trust an AI that is safer than a human driver but also more likely to crash in conditions that a human would have handled well? Time will tell.
AI brings a lot of benefits, and undoubtedly, we’ll continue to see it used in all sorts of contexts. But before you decide to add AI to your business, take a moment and ponder how you’ll respond to your fancy new tool insulting the President of China.