Original article was published by Shane Zhong on Deep Learning on Medium
If you are using a Linux system, you can leverage the power of docker to deploy the entire tool.
To start, git clone this repo locally, CD to the project directory and run the following two commands:
make build— which builds a docker image from the Dockerfile
make shell— which starts a docker container and allows you to run bash command (and thus the monitoring tool) inside the container.
Here is the detail of the above two commends, which is stored in the Makefile inside the project folder:
docker build -f Dockerfile -t "realtime-emotion-monitor" .shell: build
docker run -it -e DISPLAY=unix$(DISPLAY) -v /tmp/.X11-unix:/tmp/.X11-unix --device /dev/video0 -v $(shell pwd)/:/workspace "realtime-emotion-monitor" bash
It is slightly complex to set up the environment if you would like to run it on MacBook, since enabling Webcam inside Docker is straightforward on iOS. I would recommend running the monitoring tool via a virtual environment using PyCharm or Anaconda. Inside your virtual environment, install the following pre-requisites:
- Python 3.7 or higher
- Fast.ai library
- install libraries specified in the requirements.txt file via running the command
pip install -r requirements.txt.
- Inside the
liveVideoFrameRead.pyscript, change the variable
path = ‘/workspace’to your local project directory.
Launch the monitoring tool with a single-line
Once the environment is set up, you can run the command
python liveVideoFrameRead.py in your terminal, with the following optional tags:
—- save-videoto save webcam video to the output folder
—- save-csvto save expression predictions, their probability tensor to a csv file
—- camera_id 1to set the webcam id, default to 1 for MacBook
—- low-fpsto lower frame refresh rate to reduce the amount of real-time computation
python liveVideoFrameRead.py --save-csv --low-fps will run Webcam at low fps rate and save the prediction results to a CSV file to the output folder.
In the current logic, if the average probability of sad or angry exceeds 0.5 over the past 20 seconds (by default), it triggers an alarm sound.
def trigger_sound(sound_dir, sound_time_interval, data):
"""Function to trigger an alarm sound.
sound_dir: the directory of the alarm sound MP3 file.
sound_time_interval: the total duration of the stress-testing period in seconds, default to 20 seconds
data: the raw data which stores the emotion classification probably """
exp_list = [i for i in data]
list_angry = [i for i in exp_list]
list_sad = [i for i in exp_list]
avg_sad = sum(list_sad[-sound_time_interval:]) / sound_time_interval
avg_angry = sum(list_angry[-sound_time_interval:]) / sound_time_interval
if avg_sad >= 0.5 or avg_angry >= 0.5:
print("Trigger Sound - Relax :)")
Low FPS rate
Increasing the time gap between each inference reduces the amount of computation power required. By default, the system sleeps for 0.6 second between each inference.
# Inside the modelling loop
A pre-trained emotion classification model
The pre-trained model I used is from here. I would recommend you to take a look at it if interested. In summary, it is a fine-turned PyTorch classification model
shape_predictor_68_face_landmarks.dat , that classifies your facial expression into five emotion categories, based on the positions of 68 face landmarks.