Source: Deep Learning on Medium
Continuing on from part 1 linked below:
I’ve always been a fan of HIIT (High Intensity Interval Training) and I prefer not to heat my house in the winter…medium.com
After training a NLP model that accepts step mania files. I was able to then use the prediction mode of the NLP model such that we can then generate a foot pattern for any arbitrary song.
This exercise was less relevant to the use of deep learning and more going through the effort of creating the .dwi file that step mania expects.
This project was simplified with some libraries I was able to find that can perform beat matching and provide the offset time for the initial gap. Additionally, typical DDR songs are in the 90s to 180s range. As an excercise to the reader, a Herculean effort could be possible if one were to complete a 10+ minute long song. In fact, I would contend that they might just be the strongest person in the world if they can successfully complete Innagadadavida (17 minutes in duration).
Some of the Librosa beat matching and gap finding is shown in the code block below:
def MakeMP3(filePath, shorten = True, shortenLength = 90000):
"""Shorten the mp3 if it is greater than the length specified. Default is 90 seconds"""
song = AudioSegment.from_mp3(filePath)
newSong = song
if shorten and (len(song) > shortenLength):
newSong = song[:shortenLength]
def GetGapTime(beatTimes, gapLength = 200):
"""Gets the first beat after X milliseconds, 200 by default"""
for beatTime in beatTimes:
if beatTime > gapLength:
# Load the audio as a waveform `y`, Store the sampling rate as `sr`
y, sr = librosa.load(fileName)
tempo, beatFrames = librosa.beat.beat_track(y=y, sr=sr)
# Convert the frame indices of beat events into timestamps
beatTimes = librosa.frames_to_time(beatFrames, sr=sr)
#Convert to milliseconds
beatTimes = beatTimes * 1000
gap = GetGapTime(beatTimes)
bpm = tempo
return (bpm, gap)
The project was pretty successful. I am now able to take a folder of random mp3 and produce some reasonable DDR songs. A sample is shown in the graphic below:
And as a specific example one can see the types of patterns that it is capable of generating. I left a couple of .dwi files on the github for those that might be interested in trying this out without using the training process.
This project is reasonably successful in generating DDR songs that match the beat of the music. Overall, the patterns are arbitrarily generated without any connection to the song. Some of the songs matched very well to the music. It is essential that the bpm of the song does not change over the course of the song. It is certainly possible to change the bpm through the use of standard librosa functions and can be input into the .dwi format easily but that is beyond my interest in this project. I really was interested in being able to listen to a larger variety of music.
The most interesting idea from this project would be to create an end-to-end model that actually uses a spectrogram of a song to predict the step patterns. Librosa has the ability to produce a spectrogram easily so this would be my next area of study provided I intend to spend more time on this.
Link to the code below: