Real-time stress-level detector using Webcam

Original article was published by Shane Zhong on Deep Learning on Medium


Environment Setup

Linux System:

If you are using a Linux system, you can leverage the power of docker to deploy the entire tool.

To start, git clone this repo locally, CD to the project directory and run the following two commands:

  1. make build — which builds a docker image from the Dockerfile
  2. make shell — which starts a docker container and allows you to run bash command (and thus the monitoring tool) inside the container.

Here is the detail of the above two commends, which is stored in the Makefile inside the project folder:

build:
docker build -f Dockerfile -t "realtime-emotion-monitor" .
shell: build
docker run -it -e DISPLAY=unix$(DISPLAY) -v /tmp/.X11-unix:/tmp/.X11-unix --device /dev/video0 -v $(shell pwd)/:/workspace "realtime-emotion-monitor" bash

iOS System:

It is slightly complex to set up the environment if you would like to run it on MacBook, since enabling Webcam inside Docker is straightforward on iOS. I would recommend running the monitoring tool via a virtual environment using PyCharm or Anaconda. Inside your virtual environment, install the following pre-requisites:

  • Python 3.7 or higher
  • Fast.ai library
  • dlib
  • opencv-python
  • install libraries specified in the requirements.txt file via running the command pip install -r requirements.txt.
  • Inside the liveVideoFrameRead.py script, change the variable path = ‘/workspace’ to your local project directory.

Launch the monitoring tool with a single-line

Once the environment is set up, you can run the command python liveVideoFrameRead.py in your terminal, with the following optional tags:

  • —- save-video to save webcam video to the output folder
  • —- save-csv to save expression predictions, their probability tensor to a csv file
  • —- camera_id 1 to set the webcam id, default to 1 for MacBook
  • —- low-fps to lower frame refresh rate to reduce the amount of real-time computation

For example, python liveVideoFrameRead.py --save-csv --low-fps will run Webcam at low fps rate and save the prediction results to a CSV file to the output folder.

Algorithm personalisation

Stress-level detection

In the current logic, if the average probability of sad or angry exceeds 0.5 over the past 20 seconds (by default), it triggers an alarm sound.

def trigger_sound(sound_dir, sound_time_interval, data):
"""Function to trigger an alarm sound.
Args:
sound_dir: the directory of the alarm sound MP3 file.
sound_time_interval: the total duration of the stress-testing period in seconds, default to 20 seconds
data: the raw data which stores the emotion classification probably """
exp_list = [i[2] for i in data]
list_angry = [i[0] for i in exp_list]
list_sad = [i[3] for i in exp_list]
avg_sad = sum(list_sad[-sound_time_interval:]) / sound_time_interval
avg_angry = sum(list_angry[-sound_time_interval:]) / sound_time_interval
if avg_sad >= 0.5 or avg_angry >= 0.5:
print("Trigger Sound - Relax :)")
mixer.init()
mixer.music.load(sound_dir)
mixer.music.play()

Low FPS rate

Increasing the time gap between each inference reduces the amount of computation power required. By default, the system sleeps for 0.6 second between each inference.

# Inside the modelling loop
if args['low-fps']:
time.sleep(0.6)

A pre-trained emotion classification model

The pre-trained model I used is from here. I would recommend you to take a look at it if interested. In summary, it is a fine-turned PyTorch classification model shape_predictor_68_face_landmarks.dat , that classifies your facial expression into five emotion categories, based on the positions of 68 face landmarks.