Python program to control racing games with hand gestures

Source: Artificial Intelligence on Medium

Python program to control racing games with hand gestures

Photo by Glenn Carstens-Peters on Unsplash

I have been a big fan of gaming since childhood. Played dozens of games. The racing genre is my favorite one. So one fine day I thought to create something from the things I love the most. i.e gaming and technology.

I created a python program using OpenCV to play my once favorite NFS MostWanted with hand gestures.

Before diving into the details watch me race using my hand gestures and win the race too 😃

To get realtime results on my low-end laptop, I used a very simple principle of object detection via color thresholding(color detection). I have detected orange colors in my case. It can be changed to use any color.

Speaking theoretically, the frame is divided into two halves, left and right and each half is divided into upper and lower half. Just like 4 quadrants where presence of object in each quadrant is used to control acceleration, brake, turn-left and turn-right settings of a vehicle by simulating keypress W,A,S,D.

Had I not encountered this StackOverflow post, this program wouldn’t be possible. https://stackoverflow.com/questions/14489013/simulate-python-keypresses-for-controlling-a-game

A Gaussian Blur is applied to the image frame and is converted into HSV color space for easy thresholding by getting minimum noise.

the output of line no 9 will be a binary mask with True values on pixels that contains orange objects.

Next step would be to find contours of the objects detected for each left and right half of the frame and select the contour with the maximum area and find its centroid.

Now whenever the center of the object will move upwards or below the blue rectangle as seen in the video, a keypress will be triggered.

Full code can be found at: https://github.com/pateldigant/gesture-gaming-python

Future work:

Make use of Deep Learning to control the game by detecting some other objects, signs or via finger counting(possible with OpenCV but not too accurate).