Artificial Intelligence for Music Videos


I am doing a video analysis A.I. project for a client, and I want to share with you a super fun idea that popped into my mind tonight while I was waiting for stuff to render. I really don’t have time to be writing this, but I feel like I have to. Some ideas are just too awesome to not stay up all night programming. Let’s get an A.I. system to outline people in a music video.

There have been some great object detection articles on Towards Data Science, and so I don’t want to post duplicates of the same cool stuff. Instead of detecting the location and label of an object, lets draw lines around the object instead, using COCO and Mask_RCNN. Doing it on a music video is just extra cool.

Let’s mess with Jain’s Makeba video. I removed the audio track and kept only the first 2000 video frames, just to make it extra clear that this article is making fair use of the artist’s work for educational purposes. Enjoy her music video on YouTube by clicking here.

Here is the result of our A.I. analysis of the video:

I kind of love how it finds people in billboards and in the background. It’s really subtle. I also like how the merge of the black and white girls faces was so smooth that it faked out the A.I. into thinking it is actually one person. Here below are some cool segments of the video I thought were noteworthy. Enjoy!

It sees the finger!
It recognizes people from behind.
Lots of people? rCNN says no problemo!

Emergency Party Button

I have to add one more fun project to this post.

Data science is hard work. When we are all sitting around the office, and our model keeps overfitting, or not compiling, we need some way to get back into the success mindset. Our office culture is pretty whimsical, and so when data science is getting us down, or when we have an unexpected triumph, we slam the emergency party button. Similar products are on the market, but we wanted to make our own from scratch. Not as a product. Just for fun.

We 3D printed the parts and Matt put together the relay, raspberry pi, and other parts and wired it all up.

Here is our emergency party button demo:

We (mostly Mathieu Lemay) have been working on this project for like 2 years in our spare time. It is the side project that just won’t die. Matt added a song randomizer, relays, speakers, and really every cool bell and whistle you can imagine. It feels really good to slam that button and trigger the lights and music.

And so, in conclusion, artificial intelligence can be used to analyze video and extract a lot of information. And when it’s all too much, hit the emergency party button!

If you liked this article on A.I. applied to music videos, press the follow button, and have a look at some of my most read past articles, like “How to Price an AI Project” and “How to Hire an AI Consultant.”

Happy Coding!

-Daniel
daniel@lemay.ai ← Say hi.
Lemay.ai
1(855)LEMAY-AI

Other articles you may enjoy:

Source: Deep Learning on Medium