Face Detection, Recognition and Emotion Detection in 8 lines of code!

Source: Deep Learning on Medium

Doing cool things with data!


Humans have always had the innate ability to recognize and distinguish between faces. Now computers are able to do the same. This opens up tons of applications. Face detection and Recognition can be used to improve access and security like the latest Apple Iphone does (see gif below), allow payments to be processed without physical cards — iphone does this too!, enable criminal identification and allow personalized healthcare and other services. Face detection and recognition is a heavily researched topic and there are tons of resources online. We have tried multiple open source projects to find the ones that are simplest to implement while being accurate. We have also created a pipeline for detection, recognition and emotion understanding on any input image with just 8 lines of code after the images have been loaded! Our code is open sourced on Github.

Facial Biometric

This blog is divided into 3 parts:

  1. Facial Detection — Ability to detect the location of face in any input image or frame. The output is the bounding box coordinates of the detected faces
  2. Facial Recognition — Compare multiple faces together to identify which faces belong to the same person. This is done by comparing face embedding vectors
  3. Emotion Detection — Classifying the emotion on the face as happy, angry, sad, neutral, surprise, disgust or fear

So let’s get started!

Facial Detection

Facial detection is the first part of our pipeline. We have used the python library Face Recognition that we found easy to install and very accurate in detecting faces. This library scans the input image and returns the bounding box coordinates of all detected faces as shown below:

Face Detection

The below snippet shows how to use the face_recognition library for detecting faces.

face_locations = face_recognition.face_locations(image)
top, right, bottom, left = face_locations[0]
face_image = image[top:bottom, left:right]

Complete instructions for installing face recognition and using it are also on Github

Facial Recognition

Facial Recognition verifies if two faces are same. The use of facial recognition is huge in security, bio-metrics, entertainment, personal safety, etc. The same python library face_recognition used for face detection can also be used for face recognition. Our testing showed it had good performance. Given two faces match, they can be matched with each other giving the result as True or False. The steps involved in facial recognition are

  • Find face in an image
  • Analyze facial feature
  • Compare features for the 2 input faces
  • Returns True if matched or else False.

The code snippet that does this is below. We create face encoding vectors for both faces and then use a built in function to compare the distance between the vectors.

encoding_1 = face_recognition.face_encodings(image1)[0]

encoding_2 = face_recognition.face_encodings(image1)[0]

results = face_recognition.compare_faces([encoding_1], encoding_2,tolerance=0.50)

Lets test the model on two images below:

Face 1
Face 2

As shown on the right we have 2 faces of Leonardo Di Caprio with different poses. In the first one the face is also not a frontal shot. When we run the recognition using the code shared above, face recognition is able to understand that the two faces are the same person!

Emotion Detection

Humans are used to taking in non verbal cues from facial emotions. Now computers are also getting better to reading emotions. So how do we detect emotions in an image? We have used an open source data set — Face Emotion Recognition (FER) from Kaggle and built a CNN to detect emotions. The emotions can be classified into 7 classes — happy, sad, fear, disgust, angry, neutral and surprise.

Model — We built a 6 layered Convolutional Neural Network (CNN) in Keras and use image augmentations to improve model performance. We tried many different models and have open sourced our best implementation at this link.

You can load the pretrained model and run it on an image using only 2 lines of code below:

model = load_model("./emotion_detector_models/model.hdf5")
predicted_class = np.argmax(model.predict(face_image)


This blog demos how easy it can be to implement facial detection and recognition models in your applications. Facial detection can the starting point for many customized solutions. I hope you try our open sourced code for yourself.

I have my own deep learning consultancy and love to work on interesting problems. I have helped many startups deploy innovative AI based solutions. Check us out at — http://deeplearninganalytics.org/. If you have a project that we can collaborate on, then please contact me through my website or at info@deeplearninganalytics.org

You can also see my other writings at: https://medium.com/@priya.dwivedi