I had a spark of imagination the other day. I was singing to “Stronger Than You” by Estelle from the Cartoon Network show “Steven Universe”, a show that’s been filling the vacuum for me since “Adventure Time” ended.
I’ve not had training in several years and the pitches were terrible and it made me miss the days of playing clarinet. The notes are easier to control, almost binary. You’re either playing a pitch or you’re not.
The voice is more nuanced and can create a variety of pitches on demand and, properly trained, in ways that move us emotionally.
In Deep Learning, a branch of Artificial Intelligence, we have to rely on one of a million output functions to finish that forward propagation before the weight adjustment and ensuing back propagation. Really, anything can be an output function, but for today’s article, I’ll talk about absolute value, ReLU, and sigmoid. The best activation functions will be those that return a value between -1 and 1 or, preferably, 0 and 1.
Absolute value can do that, but it is a non-continuous function. From a calculus perspective, you cannot find the derivative of the absolute value across all input values.
This is the same for the rectified linear unit (ReLU). ReLU is better in its 0 or 1 stance and is computationally cheaper, but still not quite what you want in terms of convenient differentiation.
The sigmoid, now there’s a function. The previous functions were me on my clarinet try to get the average note between C and C# (music and programming are basically the same thing). The sigmoid is based on the best number in math (e), having an equation of (e^x/(e^x + 1)). It produces outputs between 0 and 1 and is differentiable.
It is the operatic soprano of Artificial Intelligence and I’m happy it exists.
As AI continues to solve more and more problems on a more granulated level and as more people get into it, I hope my “Steven Universe” epiphany works for you.
Jimmy Murray is a Florida based comedian who studied Marketing and Film before finding himself homeless. Resourceful, he taught himself coding, which led to a ton of opportunities in many fields, the most recent of which is coding away his podcast editing. His entrepreneurial skills and love of automation have led to a sheer love of all things related to AI.
Future of Work
Source: Deep Learning on Medium