Are Google’s AI biased to recognize black people as more “violent”?

Source: Artificial Intelligence on Medium

The game in question is called “Is this violence? Am I too sexy?” and was initially supposed to be a nonchalant game about tricking the AI into thinking violence or sexy stuff is happening in the picture. These pictures are taken in realtime through the webcam and sent to Google Cloud Vision where it returns the possibility of violence or nudity existing in the picture. The player’s goal was to maximize this score without doing actual violence or going nude.

I became good at making artificial veggie dick pics to fool an AI

Recent developments in machine learning have greatly advanced the performances of visual recognition systems that can detect people and objects from an image. Tech companies offer the ability to perform visual recognition on any images as part of their cloud services. Services such as Google Cloud Vision and IBM Watson Visual Recognition are used every day by many third-party apps. One of the functions that are used the most is detecting explicit content depicting porn or violence.

This technology is used behind Google Safe Search as well. It decides what is appropriate to show up on your image search or on social media. These functions are obviously important to prevent harmful content from getting online. but it always comes with the question — How does an AI decide subjective notions such as sexiness and violence? And would it be possible to trick it? This was the starting point for this game.

Players faking violence at Playful Arts Festival 2018

Halfway through the playtesting, we realized that there were unique patterns coming up based on who plays the game. Somehow people of darker skin color always scored higher in the violence score and people with lighter skin color always scored higher in faking the racy score. Puzzled by this, I started testing how the score changed based on the skin color by photoshopping stock photo images of people depicting obviously violent things.

According to Google’s explanation, racy content may include (but is not limited to) skimpy or sheer clothing, strategically covered nudity, lewd or provocative poses, or close-ups of sensitive body areas. And violence is pictures that are depicting killing, shooting, or blood and gore. I chose the picture of a person holding a gun because that was explained in their website of an example that scores higher as violent.

And here were the results.

*The skin color of the pictures below is changed to experiment with how an AI would see the same picture differently.

Turning dark skin color into white decreases the violence probability and increases racy score
Turning light skin color to dark results in violence scores increasing and racy scores decreasing

The results showed that there might indeed be a difference in how racy or violent a person is based on their skin color. In the experiment where I converted a lighter skin-colored person to a darker color, the violence score increased while the racy(sexiness) score decreased. On the other hand turning darker colored skinned people into a lighter color resulted in the opposite: decreased violence score and increased racy score. You can also try out the system yourself by going to Google Cloud Vision’s demo page here.