Celebrity recognition using VGGFace and Annoy

Original article was published by Shobhit Gupta on Deep Learning on Medium


Celebrity recognition using VGGFace and Annoy

This is a guide on how to create a model to recognize celebrities using a face matching algorithm.

This article includes the following topics:

  1. Guide on using the library and creating your own model.
  2. Publishing a pip package for the repository.

Refer this for detailed documentation.

All the code mentioned in this article is available on my GitHub repository.

Basic working of the model

The basic algorithm of the model includes the following:

  1. Face detection model (currently I am using MTCNN) is used to find faces in the images.

2. The cropped faces are sent to a model called VGGFace. This model provides encoding for the faces in the form of an array of size 2048.

3. The encoding are stored and indexed using a library called Annoy. This is used to find the closest matching to the faces which need to be detected.

So, in order to train your own celeb model, you need to provide a dataset of celebrities you need to recognize. Data can be scraped easily using sites like Google images, DuckDuckGo, Bing, etc. This data has to be arranged in a proper folder structure, with a separate folder for each celeb. The name of the celeb for the model is taken from the folder name (so make sure there are no spaces or special characters in the folder names). Provide as many celeb images are you can. The model will keep on improving as more data is added.

Now, you only need to provide the path of the folder in which all the folders with celeb images are stored. Details for using the code will be shown in the next section.

After the model has been created, we need to make predictions. During predictions, each images goes through the same process, face detection and encoding creation. The encoding of the face is search using the annoy index model that was created, which gives us the closest matching faces. The annoy library provides as many closest matches as we want, we can use only the top closest match for prediction. But in order to reduce false detections, a simple trick is added. We find the 10 closest matches, and if 3 or more of the matches belong to the same celeb, then the result is considered, else it is discarded.

Using the library

There are several ways to use the library:

  1. Python code files

First clone the library to your local system. Install the dependencies using the requirements.txt file:

git clone https://github.com/shobhit9618/celeb_recognition.git
cd celeb_recognition
pip install -r requirements.txt

Now, to create your own model, just open create_celeb_model.py file, provide the path to the root folder for the celeb images and run the file. This will start the model creation process and store 3 kinds of files: celeb_mapping.json for storing mapping of each celeb image (generated during training) to the celeb name, celeb_index.ann for storing indexes vs encodings for fast searching, celeb_name_encoding.pkl files for storing the generated encoding for each celeb. You can provide paths for storing each of these files in the create_celeb_utils.py files inside the celeb_utils folder.

For making predictions, provide paths to celeb_mapping.json and celeb_index.ann files in celeb_utils.py file in celeb_utils folder. Then, provide path to the images file or an image URL in the celeb_recognition.py file and run it. This will provide the predictions and the detected faces in the images. You also have the option to store the output images with the bounding boxes to your local system.

Prediction on an image of Barack Obama

2. Pip package

My pip package for the repository is available here.

To install pip package, run:

# pip release version
pip install celeb-detector
# also install additional dependencies with this (if not installed via requirements.txt file)
pip install annoy keras-vggface keras-applications
# Directly from repo
pip install git+https://github.com/shobhit9618/celeb_recognition.git

If you are using conda on Linux or Ubuntu, you can use the following commands to create and use a new environment called celeb-detector (this will install all the required dependencies):

conda env create shobhit9618/celeb-detector
conda activate celeb-detector

For using my model for predictions, use the following lines of code after installation:

import celeb_detector # on running for the first time, this will download vggface model
img_path = 'sample_image.jpg'
celeb_detector.celeb_recognition(img_path) # on running for the first time, 2 files (celeb_mapping.json and celeb_index_60.ann) will downloaded to the home directory
# if you want to use an image url, just provide the url and add url=True
url = 'https://sample/sample_image_url.jpg'
celeb_detector.celeb_recognition(url, url=True)

This returns a list of dictionaries, each dictionary contains bbox coordinates, celeb name and confidence for each face detected in the image (celeb name will be unknown if no matching face detected).

For using your own custom model, also provide path to json and ann files as shown below:

import celeb_detector
img_path = 'sample_image.jpg'
ann_path = 'sample_index.ann'
celeb_map = 'sample_mapping.json'
celeb_detector.celeb_recognition(img_path, ann_path, celeb_map)

For creating your own model and run as follows:

import celeb_detector
folder path = 'celeb_images'
celeb_detector.create_celeb_model(folder path)

3. Jupyter notebook

  • Provide paths to celeb_mapping.json and celeb_index.ann files in celeb_recognition.ipynb file. If you want to try my model, ignore this step.
  • Run all the cells in the celeb_recognition.ipynb file, the final cell will provide widgets for uploading images and making predictions (this will also download the necessary model files).
  • NOTE: celeb_recognition.ipynb is a standalone file and does not require any other files from the repo for running.
  • You can also open and use the file in google colab. Just uncomment the first cell and run it for installing all the requirements.
  • You can also run a binder application by clicking here.
  • You can also launch a voila binder application (which only has widgets for image upload and celeb prediction) by clicking here.

Publishing a pip package for the repository

Making your own pip package is surprising easy, provided your code is clean and build using appropriate functions and classes.

General guide for creating a pip package can be found on the official documentation at:

The basic requirements are to create a LICENSE file, setup.py file and a folder which should have all the codes to be included in the pip package.

Python setuptools package can directly find all the necessary functions that are to be included in the pip package. You also need to make an account on the PyPI sites and get your token for publishing the package.

In my case, I have added a folder called celeb-detector (which is the name of my pip package). Inside this folder has files which include functions for getting image predictions and creating the model. Once the pip package is installed and imported, you can use the functions to get predictions or create your own model.